FootnotelyDoes Cloudflare block GPTBot and other AI crawlers?
Sometimes, and it depends on your settings, when your domain was added, and which bot you mean. Here's what Cloudflare's own docs say, how to check your site, and how to stay visible in AI search while still saying no to training.
Updated October 19, 2026 · By the Footnotely team
The short answer
Cloudflare doesn't block every AI bot for every site. But it gives you settings that can, and some sites have them on without realising. Since September 15, 2026, Cloudflare splits AI traffic into three behaviours you control separately (Cloudflare docs):
- Search: crawlers that index content to answer questions later.
- Agent: real-time fetches on a person's behalf (chat fetch bots, browser agents).
- Training: crawlers collecting content to train models, including mixed-purpose bots.
Each can be set to block on all pages, block only on pages with ads, or allow.
What the defaults are
According to Cloudflare, new domains added from September 15, 2026 get Training and Agent bots blocked on pages that display ads, with Search allowed. Cloudflare's announcement says sites without ads get all three allowed by default. Older domains keep whatever they had. If someone switched on the legacy "Block AI bots" toggle, training crawlers like GPTBot were already blocked.
Why it matters for AI visibility
GPTBot is OpenAI's training crawler. ChatGPT's search results rely on OAI-SearchBot, and live user fetches come from ChatGPT-User (see OpenAI's bot docs). If the Agent setting is set to block, then "browse this page" style requests can fail. If Search is set to block, your pages can't be indexed for AI answers. Plenty of SaaS founders want to refuse training but still be found. That setup is possible, but only if you leave Search (and, ideally, Agent) allowed.
How to check your site
- In the Cloudflare dashboard, go to Security Settings → Configure AI bot policies and read the three settings.
- Open
yourdomain.com/robots.txt. Look for a "Cloudflare Managed Content" block with lines likeUser-agent: GPTBot / Disallow: /. - Test real access, not just robots.txt: our free AI crawler checker requests your page as GPTBot, OAI-SearchBot, PerplexityBot and others and shows the status each one gets.
A sensible setup for most SaaS sites
- Search: Allow.
- Agent: Allow (your pricing and docs pages usually have no ads anyway).
- Training: your call. Blocking it shouldn't affect search crawlers that operators run separately.
For per-bot rules, see robots.txt for AI crawlers. Once access is fixed, check whether AI answers actually mention you. Being crawlable is necessary, but it doesn't guarantee you get mentioned.
FAQ
Does blocking GPTBot remove me from ChatGPT search?
Not by itself. OpenAI documents GPTBot as its training crawler and OAI-SearchBot as the crawler used for search results. Blocking one is a separate decision from blocking the other. Check both.
Is Cloudflare's managed robots.txt the same as a block?
No. Cloudflare's docs note robots.txt compliance is voluntary; it states a preference. Actual blocking is done by the bot policies / AI Crawl Control, which can return an error to the crawler.
Is this on the Free plan?
Per Cloudflare's changelog, the Search / Agent / Training controls and managed robots.txt are available to all plans, including Free.
Track it every week instead of checking by hand
Footnotely asks your buyer questions on ChatGPT, Perplexity and Gemini several times per run and shows the mention rate with its sample size, the raw answers, and who gets named instead of you. Free plan, then from $29/mo.