FootnotelyIs GPTBot blocked on your site? Free AI crawler checker
Enter your domain. We read your robots.txt and fetch your homepage as 12 AI crawlers, including GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot, to see if anything blocks them. Free, no signup.
Updated October 19, 2026 · By the Footnotely team
What we check
- robots.txt rules for each AI user-agent, using the standard matching rules in RFC 9309.
- Live access: we request your homepage with each crawler's user-agent and look for 403s, challenges and blocks.
- Other signals: llms.txt, JSON-LD structured data, noindex tags, and how much text is server-rendered (many crawlers don't run JavaScript).
The crawlers, and who documents them
| User-agent | Used for | Docs |
|---|---|---|
| GPTBot | OpenAI model training | OpenAI |
| OAI-SearchBot | ChatGPT search results | OpenAI |
| ChatGPT-User | Pages fetched when a user asks | OpenAI |
| PerplexityBot / Perplexity-User | Perplexity index and live fetches | Perplexity |
| Googlebot / Google-Extended | Search + AI Overviews / Gemini training opt-out | |
| ClaudeBot / Claude-SearchBot | Anthropic training / search | Anthropic |
| CCBot | Common Crawl open dataset | Common Crawl |
Want copy-paste rules? See robots.txt for AI crawlers.
FAQ
Does blocking GPTBot remove me from ChatGPT search?
Not by itself. OpenAI documents GPTBot as its training crawler and OAI-SearchBot as the crawler for ChatGPT search results. They are controlled separately in robots.txt.
My robots.txt allows everything but the check says blocked. Why?
Your CDN or firewall may be blocking bot user-agents, for example a bot-fight or AI-scraper blocking setting. Check your CDN's bot settings.
Is this check the same as what the real bots see?
Close, not identical. We test robots.txt rules exactly and fetch with each bot's user-agent, but some CDNs verify real crawlers by IP address and may treat them differently.
Track it every week instead of checking by hand
Footnotely asks your buyer questions on ChatGPT, Perplexity and Gemini several times per run and shows the mention rate with its sample size, the raw answers, and who gets named instead of you. Free plan, then from $29/mo.