AI Crawler List 2026: Every AI User-Agent and What It Does

AI crawler list: every AI user-agent and what it does

AI companies run three kinds of bots: search crawlers that decide whether you can be cited, user-triggered agents that fetch a page when someone asks about it, and training crawlers that collect data for future models. You can allow one kind and block another. Here is each user-agent, sorted by what it does.

Updated 2026-10-19

Search & citations crawlers

Blocking these removes your pages from that product’s search results and AI answers. Most sites should allow them.

User-agent token	Vendor	What it does	Docs
`Googlebot`	Google	Google Search crawler	Official
`Bingbot`	Microsoft	Bing Search crawler	Official
`OAI-SearchBot`	OpenAI	ChatGPT Search indexing and citations	Official
`PerplexityBot`	Perplexity	Perplexity Search indexing and sourced answers	Official
`Claude-SearchBot`	Anthropic	Claude search indexing	Official
`Applebot`	Apple	Apple Search / Spotlight crawler	Official
`Amazonbot`	Amazon	Amazon product and site crawler	Official

User-triggered fetch crawlers

These fetch a specific page because a user pasted a link or asked an assistant to read it. They are not bulk crawlers.

User-agent token	Vendor	What it does	Docs
`Google-CloudVertexBot`	Google	Vertex AI user-triggered site fetch	Official
`ChatGPT-User`	OpenAI	User-initiated fetches from ChatGPT	Official
`Perplexity-User`	Perplexity	User-initiated fetches from Perplexity	Official
`Claude-User`	Anthropic	User-initiated fetches from Claude	Official

Model training crawlers

These collect content that may be used to train models. Blocking them does not affect search visibility.

User-agent token	Vendor	What it does	Docs
`Google-Extended`	Google	Gemini / Google AI training and grounding control	Official
`GPTBot`	OpenAI	OpenAI foundation-model training crawler (not search)	Official
`ClaudeBot`	Anthropic	Anthropic crawler for model training	Official
`Applebot-Extended`	Apple	Apple Intelligence training and grounding control	Official
`CCBot`	Common Crawl	Common Crawl open web crawl (often used for training corpora)	Official
`meta-externalagent`	Meta	Meta AI training crawler	Official

Retired user-agents

Still seen in old robots.txt files. Rules for them do nothing for the current crawlers.

Claude-Web — Legacy Anthropic agent — superseded by Claude-User / Claude-SearchBot

robots.txt recipes

Be cited in AI search, opt out of training

User-agent: *
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
Disallow: /

Opt out of ChatGPT entirely

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
Disallow: /

Remember that a crawler named in its own group ignores your User-agent: * rules. Build a complete file with the robots.txt generator.

When robots.txt is not enough

Your CDN or firewall answers before robots.txt is read. Cloudflare has blocked known AI crawlers by default on new domains since July 2025, and many hosts rate-limit unfamiliar bots. If a crawler is allowed in robots.txt but your logs show 403 or 429 responses for it, change the CDN rule. The AI crawler checker shows both layers side by side.

Frequently asked questions

Which AI crawlers should I allow?

Allow the search crawlers of the products you want to be cited in — OAI-SearchBot for ChatGPT Search, PerplexityBot, Claude-SearchBot, and Googlebot and Bingbot for classic search and AI Overviews. Whether to allow training crawlers such as GPTBot, ClaudeBot and Google-Extended is a separate business decision.

What is the difference between GPTBot and OAI-SearchBot?

GPTBot collects data that may be used to train OpenAI’s models. OAI-SearchBot indexes pages so they can be shown and linked in ChatGPT Search. OpenAI documents them as independent settings.

Do user-triggered agents like ChatGPT-User follow robots.txt?

These agents fetch a page because a person asked for it, so vendors treat them differently from automated crawlers. Perplexity documents that Perplexity-User generally ignores robots.txt for that reason. If you must keep content away from them, use authentication rather than robots.txt.

How can I tell if a request really comes from an AI crawler?

User-agent strings are easy to fake. Google, Bing, OpenAI, Perplexity and others publish IP ranges or reverse-DNS methods for verification. Check the request IP against the vendor’s published list before trusting the user-agent.

Does blocking AI crawlers hurt my Google rankings?

No. Google Search uses Googlebot; blocking GPTBot, ClaudeBot or even Google-Extended has no effect on Google Search ranking. Blocking Googlebot itself removes you from Google Search, including AI Overviews.