How the generated file works
The output uses two groups. User-agent: * holds the rules for everyone, including your private paths. Crawlers you block share a second group that ends in Disallow: / — several User-agent lines in one group is valid under RFC 9309, the robots.txt standard Google documents its crawlers against.
Allowed crawlers are deliberately not listed. If they were, each would need its own copy of your private-path rules, because a crawler that finds a group with its name ignores the * group. Leaving them out is shorter and safer.
Which preset should you use?
- AI search yes, training no — for businesses, publishers and stores that want to be cited and clicked from AI answers but don’t want to supply training data. Most sites.
- Allow everything — for documentation, open-source projects and anyone who wants maximum reach, including being part of future model knowledge.
- Block all AI — keeps classic Google, Bing and Apple search but removes you from ChatGPT, Claude and Perplexity answers. Note that Google AI Overviews still use Googlebot.
Mistakes this generator avoids
- Blocking Google-Extended to leave AI Overviews. It doesn’t; AI Overviews follow Googlebot rules.
- An empty
Disallow:line. It means “allow everything”, not “block everything”. - Blocking CSS and JavaScript. Google needs them to render the page; never disallow your asset folders.
- Wrong case in paths. Paths are case-sensitive:
/Admin/does not block/admin/. - Using robots.txt for secrets. The file is public. Protect private areas with authentication.
Test it after uploading
Run the AI crawler checker on any page of your site. It reports, for each crawler, which rule matched and whether a CDN or firewall answered the request differently. In Google Search Console, the robots.txt report shows the version Google last fetched.