Training or citations: what your robots.txt says to eleven AI crawlers
robots.txt is one file, but its visitors read it separately: OpenAI's GPTBot trains models, OAI-SearchBot fetches citations, Perplexity-User fetches on request. The robots and sitemap tool judges every agent from a fixed roster of eleven — OpenAI, Anthropic, Perplexity, Google, Apple, Common Crawl and Meta — separately on your homepage, by the same rules the sitemap cross-check already used.
The AI layer measures, it does not advise: per agent it records what your file says to him — allowed or refused, and which group that follows from. llms.txt is measured too, at /llms.txt and /.well-known/llms.txt, with content type: a catch-all that returns html on the map path is a measured fault.
Start at your own address: who may read your homepage — only to cite, or also to train? The tool lays bare, agent by agent, what your policy really says; the choice remains yours.
1. Training, search, on request: three kinds of visitor
There used to be one crawler and one letter for everyone. Since AI agents are separated, that letter is addressed to each recipient separately: GPTBot and CCBot gather training material, OAI-SearchBot and Claude-SearchBot fetch texts to cite and search, ChatGPT-Agent and Perplexity-User fetch a page because a person asked for it. A Disallow that shuts out the first kind says nothing about the other two — and the tool reports, per agent, which of the three roles he plays and what applies to him.
Two names in the roster are not a crawler string but a control token in robots.txt: Google-Extended and Applebot-Extended. They are on the roster because the file itself addresses them to its visitors — the measurement reads what stands there, not what a publisher eventually does with it.
2. The same grammar, a different guest
The AI verdicts are not a second opinion: they fall out of the same RFC 9307 rules the cross-check already used — the longest matching rule wins, a Disallow without a valid path forbids nothing, and the wildcard group inherits to every agent the file does not name. If there is a group of its own for GPTBot with Disallow: /, that is what stands before GPTBot alone; the search agents of the same house simply walk through the wildcard gate.
That is why "blocking all AI" is no longer a single choice: a User-agent: * with Disallow: / shuts out every agent, including the agents that bring visitors and citations. The tool reports both cases as a choice you made — naming the agents — and not as a fault; what you want to allow is policy, what stands there is measurement.
3. llms.txt: a map, not a door
Beside robots.txt a second document has been rising since 2024: llms.txt, a plain-text map of the site that some agents read voluntarily. Digizap measures it at both standard spots, /llms.txt and /.well-known/llms.txt, and reports status and content type. A map that stands there is a fact worth naming; an html catch-all on the map path — what single-page apps often produce — is a measured fault: 200 at the address, but no map.
llms.txt replaces nothing: robots.txt remains the door that honest crawlers obey, the map is an offer. Just as little as the index rules in head and header take the place of robots.txt, the rule holds: what the map says, only holds for those who want to read it.