Footnotelyrobots.txt for AI crawlers: what each bot does and what to allow
Most AI companies run more than one crawler: one for training, one for search answers, one for live fetches. Blocking the wrong one can quietly keep you out of AI answers. Here's what each does, with links to each company's own docs.
Updated October 19, 2026 · By the Footnotely team
Training bots vs search bots
The key split: training crawlers collect data to build future models. Search and user crawlers fetch pages so an assistant can cite them in an answer right now. If you want to be recommended, the second group matters most.
| Company | Training | Search / live answers | Source |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot, ChatGPT-User | docs |
| Perplexity | (n/a) | PerplexityBot, Perplexity-User | docs |
| Google-Extended (token) | Googlebot (Search + AI Overviews) | docs | |
| Anthropic | ClaudeBot | Claude-SearchBot, Claude-User | docs |
| Common Crawl | CCBot (open dataset) | (n/a) | docs |
Vendors rename and add bots. Check the linked docs before relying on this table.
Example: welcome AI search
# Allow AI search and assistants to read the site User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Claude-SearchBot Allow: / User-agent: * Disallow: /app/ Sitemap: https://yourdomain.com/sitemap.xml
Example: stay in AI search, skip training
# Stay in AI search, opt out of model training User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: CCBot Disallow: / User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: PerplexityBot Allow: /
Note: listing several User-agent lines above one set of rules applies those rules to all of them (RFC 9309). A bot follows only the most specific group that names it, so a named group overrides User-agent: *.
The silent blocker: your CDN
Your robots.txt can say "allow" while your CDN or firewall returns a 403 or a challenge page to bots. Bot-protection features on CDNs can block AI crawlers by default. Run the free AI crawler checker to test both at once.
FAQ
Can I opt out of AI training but stay in AI search?
For OpenAI, yes per its docs: disallow GPTBot and allow OAI-SearchBot. Google-Extended controls use of your content for Gemini training and grounding without affecting Google Search, per Google's docs. Read each vendor's docs for current details.
Does robots.txt actually stop bots?
It's a voluntary standard. The major vendors say their crawlers respect it, but it is not access control. Use your CDN or firewall for hard blocks.
Where does robots.txt go?
At the root of your domain: https://yourdomain.com/robots.txt. Each subdomain needs its own.
Track it every week instead of checking by hand
Footnotely asks your buyer questions on ChatGPT, Perplexity and Gemini several times per run and shows the mention rate with its sample size, the raw answers, and who gets named instead of you. Free plan, then from $29/mo.