The three checks, in order
1. robots.txt allows the search crawler
ChatGPT Search uses OAI-SearchBot. Perplexity uses PerplexityBot. Claude’s search uses Claude-SearchBot. Google AI Overviews use Googlebot. Each one reads only its own group in robots.txt, or the * group if it has none (RFC 9309).
GPTBot is not in that list. It collects text for training. Blocking it does not remove a page from ChatGPT Search, and allowing it does not put a page there. The split is documented by OpenAI, and the same pattern exists for Google-Extended vs Googlebot and ClaudeBot vs Claude-SearchBot. Details: should you block GPTBot?
2. The CDN returns the page, not a challenge
Bot-protection products often allow Googlebot (they can verify its IPs) and challenge everyone else. OAI-SearchBot and PerplexityBot then receive a 403 or an interstitial, so there is nothing to cite. robots.txt will still say “allowed”. The only way to see this is to request the URL with that user-agent.
Treat the result as a hint. User-agent strings are trivial to fake, and a CDN may allow the vendor’s real IP ranges while blocking an audit server that only copies the user-agent. OpenAI, Google, Bing and Perplexity publish how to verify their crawlers; use that before you conclude a real bot is blocked.
3. The answer is in the HTML the server sends
OAI-SearchBot, PerplexityBot and ClaudeBot fetch HTML and do not run the page’s JavaScript. A product grid, a price, or a paragraph that appears only after React hydrates is an empty page to them. Googlebot is the exception: it renders. That is why a URL can rank in Google and never appear in ChatGPT or Perplexity.
What a citable passage looks like
Once the crawler can read the page, it still has to lift a passage. The passages that get reused have a shape:
- The first paragraph defines the thing in one or two sentences: what it is, who it is for, and the concrete fact.
- Headings are the questions people ask, and the first sentence under each heading answers it.
- Steps are a numbered list. Comparisons are a table. Both survive quotation better than a paragraph of adjectives.
- Numbers have units and a source. “Faster” is not citable. “Most audits finish within 20 seconds” is.
- The organization is named in the text and in JSON-LD, with
sameAslinks to profiles that actually exist.
There is no special “AI schema”. FAQ, Article and Organization markup help machines connect names to things. Markup that describes reviews, prices or authors the page does not show is a guideline violation, and it is a bad idea even when a validator stays quiet.
Why Perplexity and ChatGPT disagree
They do not share an index. A site can be in one and absent from the other for a boring reason: one bot is allowed and the other is not. Check the two rows separately. Perplexity-User and ChatGPT-User are a third case — they fetch a URL because a person asked, and Perplexity documents that Perplexity-User may ignore robots.txt. They are not how you get into the standing index.
| ChatGPT Search | Perplexity | Google AI Overviews | |
|---|---|---|---|
| Index crawler | OAI-SearchBot | PerplexityBot | Googlebot |
| On-demand fetch | ChatGPT-User | Perplexity-User | Googlebot |
| Training control | GPTBot | None separate | Google-Extended |
| Renders JavaScript | No | No | Yes |
What to do this week
- Allow OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot and Bingbot. Block training crawlers only if that is the policy you want — see the robots.txt generator.
- Allow those same user-agents in the CDN, or turn the challenge off for them.
- View the page with JavaScript disabled. If the answer is gone, render it on the server.
- Rewrite the opening paragraph as a definition, and put one sourced number in it.
- Add Organization JSON-LD with a real logo and real
sameAsURLs. - In analytics, segment referrals from chatgpt.com and perplexity.ai so you can see whether the change did anything. Rankings and citations also depend on other sites mentioning you, which no on-page edit can fake.
Run the URL through the audit or the AI crawler checker. Every failed check shows the evidence (the robots.txt line, the HTTP status, or the missing paragraph) and a fix you can paste. Most audits finish within 20 seconds, with a limit of 20 audits per 10 minutes. Google stops reading robots.txt after 500 KiB, so a short allow-list is nowhere near that cutoff.