Search crawler and training crawler are different products
Each vendor publishes two (sometimes three) user-agents. One fetches pages so an answer can cite them. Another collects text for training. A third fetches a page only when a person pastes a link or asks about it. OpenAI, Google, Anthropic and Apple document the split. The settings are independent.
| Vendor | Keep for citations | Optional to block (training) | Official docs |
|---|---|---|---|
| OpenAI | OAI-SearchBot, ChatGPT-User | GPTBot | OpenAI bots |
| Googlebot | Google-Extended | Google crawlers | |
| Anthropic | Claude-SearchBot, Claude-User | ClaudeBot | Anthropic crawler |
| Apple | Applebot | Applebot-Extended | About Applebot |
| Perplexity | PerplexityBot, Perplexity-User | No separate training bot | Perplexity bots |
Perplexity does not publish a training crawler you can block while staying in Perplexity answers. Blocking PerplexityBot opts you out of Perplexity’s index. The full list, including Common Crawl and Meta, is on the AI crawler list.
A robots.txt that blocks training and keeps search
Under RFC 9309, a crawler uses the group with its own user-agent if one exists, and ignores the User-agent: * group. So every training bot belongs in one group, and that group is the only place Disallow: / appears. OAI-SearchBot has no group of its own, so it follows * and is allowed.
User-agent: *
Allow: /
# AI training crawlers (does not affect AI search or Google Search)
User-agent: Google-Extended
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
Disallow: /
Sitemap: https://example.com/sitemap.xml
Replace the sitemap URL. The same file is what the robots.txt generator builds on the “allow search, block training” preset. Do not add a separate User-agent: GPTBot block later in the file with a different rule — the most specific matching group wins, and a second group replaces this one for that bot.
What blocking does not do
- It does not delete pages a model already trained on. robots.txt stops future fetches. It does not edit a shipped model.
- It does not hide a page from a person using ChatGPT. ChatGPT-User fetches a URL someone asked about. Treat private content with authentication.
- It does not fix a CDN block. If your firewall challenges every non-browser user-agent, OAI-SearchBot never sees the page even when robots.txt allows it.
- It is not a ranking boost. Allowing GPTBot is not a documented ranking factor for Google or for ChatGPT Search.
How to check the result
- Publish the file at
/robots.txton the same host as the pages (a CDN hostname needs its own file). - Confirm GPTBot is disallowed and OAI-SearchBot, PerplexityBot, Claude-SearchBot and Googlebot are allowed for the URL you care about.
- Request the URL with those user-agents. A 403 or a JavaScript challenge means the firewall, not robots.txt, is the block.
The free audit does both: it parses robots.txt per RFC 9309 (and shows the line that matched) and sends a live request as each crawler. A live request from an audit server is a hint — real crawlers come from published IP ranges, and a CDN can treat those differently. Most audits finish within 20 seconds, with a limit of 20 audits per 10 minutes. Google stops reading a robots.txt file after 500 KiB; the block above is a few hundred bytes.