robots.txt for AI agents

Most robots.txt files on the web were written for search engines and have not been revisited since. That was harmless while crawling meant indexing. It stopped being harmless when the same file started deciding whether an assistant can cite you.

The agents worth naming

These are not interchangeable, and the distinction that catches people out is between the crawler that gathers pages in bulk and the fetcher that retrieves one page because a user asked a question right now.

Longest match wins, and Allow breaks the tie

Google and Bing resolve rules by specificity, not by order: the longest matching pattern wins, and an Allow of equal length beats a Disallow. This is what makes it possible to open a single path inside a blocked directory.

User-agent: GPTBot
Allow: /api/mcp
Allow: /.well-known/
Disallow: /api/

We see the unfixed version constantly: a site publishes a server card at /.well-known/mcp.json pointing at /api/mcp, and a years-old Disallow: /api/ hides the endpoint from every well-behaved crawler. The endpoint works perfectly for anyone who already knows the URL, and is invisible to everything that would have discovered it. Read the discovery guide for the whole pattern.

A file worth having

# Search
User-agent: Googlebot
Allow: /
Disallow: /admin/

User-agent: Bingbot
Allow: /
Disallow: /admin/

# AI crawlers and live fetchers
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: Google-Extended
Allow: /
Allow: /.well-known/
Disallow: /admin/
Disallow: /cart/
Disallow: /account/

# Everything else
User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

Consecutive User-agent lines form one group, which keeps the file readable. Do not disallow /api/ wholesale if anything under it is meant to be found.

Why we mark the catch-all as partial

A site whose only rule is User-agent: * with Allow: / does permit every AI crawler. Our check reports that as partial rather than a pass, and the reason is not pedantry: it means nobody has made a decision. The next person to add a blanket Disallow for an unrelated reason will silently take out the assistants too, and nothing in your analytics will show it.

Naming the agents is a small act of documentation. It says we considered this, and it makes the next change deliberate.

robots.txt is a request, not a control

Compliant crawlers obey it. Your WAF, rate limiter and bot management do the actual enforcing — and they routinely block user agents your robots.txt welcomes. That mismatch is why we test with real agent user agents as a separate check: the permission you wrote is worth nothing if the edge returns 403.

If you want to state usage terms rather than block outright, content signals in robots.txt let you express intent — search yes, training no, for instance — without removing yourself from the answers.