Is Cloudflare blocking AI bots you allowed in robots.txt?

You allowed OAI-SearchBot in robots.txt, yet the bot may still not get in. The request meets your CDN or security layer first, and robots.txt comes later. This guide uses Cloudflare as the example and shows what to check.

Is a robots.txt rule the same as a CDN rule?

No. robots.txt is a request: "this bot may go here, not there." Compliance is voluntary. Cloudflare's docs say so directly: robots.txt compliance is voluntary and the file does not technically stop crawlers from reaching content.

Enforcement comes from Cloudflare's security rules. So there are two layers:

If robots.txt says "allow" but Cloudflare says "block", the rule at the door wins in practice. The bot may read an "allow" and still be refused. We could not find how Cloudflare's docs resolve a direct conflict page by page; test it on your own site rather than assume.

How does Cloudflare classify AI bots?

Cloudflare does not use one "AI bot" label. It classifies bots by behavior. The three behaviors in its docs:

• Search: indexes content to answer questions later.
• Agent: acts in real time on a person's behalf, such as chat fetch bots and browser-use agents.
• Training: crawls content to train or fine-tune a model.

One bot can have more than one behavior. Per the docs, AI search crawlers have been classified as Search since 1 July 2026.

Each category has three options: block on all pages, block on pages with ads, or allow. The setting is under Security Settings > Configure AI bot policies.

What is the 15 September 2026 default?

According to Cloudflare's docs, from 15 September 2026 new domains block Training and Agent bots on pages that show ads, while Search stays allowed. Mixed-purpose crawlers (both training and search) are blocked under every training configuration. The legacy "Block AI bots" setting is also deprecated on 15 September 2026.

The docs do not state the default for existing domains: not verified. Do not assume you are fine; look at your own settings.

One detail: the Agent category can cover bots that fetch a page because a user asked. For those, user-triggered bots such as ChatGPT-User and Perplexity-User are a separate topic; a separate guide covers how they treat robots.txt and how to protect a private page.

Cloudflare's managed robots.txt

Cloudflare offers a feature that generates robots.txt content from your policies. Per the docs:

• It is not on by default. It works when you set Training to Disallow and turn on Bot Preference Sync.
• It adds Disallow lines for known AI crawlers such as GPTBot, ClaudeBot, CCBot and Bytespider.
• If you already have a robots.txt, Cloudflare prepends its lines to your file. If not, it creates one.
• With Training set to Disallow, edge blocking for training crawlers is also enabled (mixed-use crawlers excluded).

So the robots.txt a bot sees may differ from the file on your server. Open yourdomain.com/robots.txt in a browser to see what bots are served.

What does AI Crawl Control do?

In AI Crawl Control you can allow or block each AI crawler. Per the docs:

• Choosing "block" creates or updates a WAF custom rule on your zone.
• On paid plans you can return 403 or 402.
• The table shows how many times each crawler has violated your robots.txt.
• On the free plan, detection relies on the user-agent strings of self-identifying crawlers.
• A failed request may not come from this section; any other rule or an error can cause it.

Choosing "allow" does not guarantee that another WAF rule will not still block the bot; the docs do not describe that interaction.

Checklist

1. Open yourdomain.com/robots.txt. Are there Cloudflare lines?
2. In Cloudflare, open the AI bot policies under Security > Bots. Is Search set to allow?
3. In AI Crawl Control, are the bots you allowed set to block?
4. Do any WAF custom rules, country rules or IP limits cut these bots off?
5. In your server or Cloudflare logs, is there a 403 for that bot's user-agent? To tell real visits from fake ones, use the IP lists the companies publish: OpenAI publishes a separate JSON list per bot, and so does Perplexity.
6. After a change, allow time. OpenAI's and Perplexity's docs say changes can take up to 24 hours to reflect.

To explicitly allow a bot in Cloudflare, Perplexity's docs suggest a custom rule that matches the user-agent and the published IP ranges together, with the action set to Allow. The same docs say to keep the IP ranges current from the official JSON endpoints and to watch the logs after any change. The user-agent alone is not enough, because it is easy to fake.

What does Ruescan show here?

Ruescan, our site scanner, is not open yet (the early-access list is open on the home page). When it opens, it will show which AI bots your robots.txt allows, blocks or does not mention. That answers "what does the file say". Whether your CDN or firewall actually lets the bot in is a separate question; use the checklist above in your own dashboard.

Frequently asked questions