SchemaCraft.AI
عربي

Free tool · No sign-up

AI crawler checker

Find out in one run whether ChatGPT, Claude, Perplexity, Gemini and Google are allowed to crawl a page — per crawler, with the exact robots.txt line that decides it — and whether the content is readable without JavaScript.

Updated

What the checker tests

  • robots.txt, per crawler. Rules are evaluated the way crawlers apply them under RFC 9309: the most specific user-agent group wins, the longest matching path wins, and Allow wins a tie. A crawler with its own group ignores the User-agent: * group completely.
  • Page-level blocks. noindex in meta robots or the X-Robots-Tag header, and non-200 responses.
  • JavaScript dependence. OpenAI’s and Anthropic’s crawlers download HTML but do not execute JavaScript, so a page that renders its text in the browser is close to empty for them. We show exactly what a non-JavaScript crawler reads.
  • Live requests. The page is requested again with each major crawler’s user-agent so you can see CDN or firewall responses — reported as a hint, for the reason in the FAQ below.
  • llms.txt and sitemap. Whether they exist, are served as text, and are referenced correctly.

Search crawlers and training crawlers are different bots

Every major AI company now runs separate user-agents for search and for model training. Blocking training is a business decision; blocking search crawlers removes you from that product’s answers. The audit does not penalize blocked training crawlers.

VendorSearch / answersUser-triggeredTraining
OpenAIOAI-SearchBotChatGPT-UserGPTBot
AnthropicClaude-SearchBotClaude-UserClaudeBot
PerplexityPerplexityBotPerplexity-User—
GoogleGooglebot (incl. AI Overviews)Google-CloudVertexBotGoogle-Extended
AppleApplebot—Applebot-Extended

The full list with official documentation links is in the AI crawler list.

The most common reasons AI crawlers are blocked

1. A bot-specific group that forgets the defaults

Adding User-agent: GPTBot with Allow: / means GPTBot no longer reads your * group — your private-path disallows stop applying to it. Repeat the disallows inside every named group, or don’t name crawlers you simply want to allow.

2. A CDN or firewall rule

Cloudflare’s AI crawler blocking, “bot fight” modes and hosting-level rate limits often return 403 or 429 to AI crawlers while Googlebot gets through. If robots.txt allows a crawler but your logs show errors for it, the fix is in your CDN settings, not in robots.txt.

3. Content that only exists after JavaScript runs

Single-page apps that ship an empty <div id="root"> are readable by Googlebot (which renders) but not by most AI crawlers. Server-side rendering or static generation fixes this.

Allow AI search, block training

This is the setup most publishers and businesses choose. Paste it into your robots.txt and keep your existing rules above it:

User-agent: *
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
Disallow: /

Or build your own with the robots.txt generator.

Frequently asked questions

How do I check if my site blocks ChatGPT?

Enter your URL above. The checker reads your robots.txt and evaluates the rules for OAI-SearchBot (ChatGPT Search), ChatGPT-User (links users open from ChatGPT) and GPTBot (training) separately, using the same matching rules the crawlers use. It also fetches the page and reports whether the content is in the raw HTML, because ChatGPT’s crawlers do not run JavaScript.

Does blocking GPTBot remove my site from ChatGPT search?

No. GPTBot collects training data. ChatGPT Search uses OAI-SearchBot, and OpenAI documents the two as independent controls. You can block GPTBot and still appear in ChatGPT Search as long as OAI-SearchBot is allowed.

Does blocking Google-Extended remove me from AI Overviews?

No. Google-Extended only controls whether your content is used for Gemini model training and grounding outside Search. AI Overviews and AI Mode are part of Google Search and use Googlebot, so they follow your Googlebot rules.

robots.txt allows the bot but it still cannot reach my site. Why?

robots.txt is only a request. CDNs and firewalls can block crawlers before robots.txt is ever read — Cloudflare, for example, blocks known AI crawlers by default on new domains since July 2025. Check your CDN’s bot settings and your server logs for 403 or 429 responses to the crawler’s user-agent.

Why does the report say the live bot test is “not verifiable”?

We can send a request with a crawler’s user-agent, but we cannot send it from that vendor’s IP addresses. Firewalls that verify crawler IPs correctly block impostors, so a blocked test request does not prove the real crawler is blocked — and an allowed one does not prove it is allowed. The robots.txt verdict is exact; the live test is a hint.

Related free tools

robots.txt generatorAllow AI search crawlers, block training crawlers.AI crawler listEvery AI user-agent, its purpose and official docs.llms.txt generatorBuild a valid llms.txt file from your key pages.Full AI search auditAbout 50 SEO, GEO and AEO checks on any URL, with evidence and fixes.