[Writing](https://ruesandora.com/writing/) Guide

# If ChatGPT-User ignores robots.txt, how to protect a page?

By [Rues](https://ruesandora.com/about) · Published 19 October 2026 · Updated 19 October 2026

Sources checked on 16 October 2026

Short answer

Do not protect a private page with robots.txt. OpenAI says robots.txt rules may not apply to ChatGPT-User because a user starts the action; Perplexity says Perplexity-User generally ignores robots.txt. Anthropic says Claude-User follows robots.txt. For a page that must stay private, use a password or login; to keep it out of search, use noindex.

A user tells ChatGPT "read this address", and ChatGPT-User fetches your page, even if you disallowed it in robots.txt. This guide covers what each company says and how to protect a page that really must stay private.

## What is a user-triggered bot?

AI company bots do one of three jobs: index for search, collect for model training, or fetch a page at that moment to answer a user's question. The third kind is a user-triggered bot. It does not roam on its own; it runs because a person asked.

## What does each company say?

| Bot | Statement in the company's docs | robots.txt |
| --- | --- | --- |
| ChatGPT-User (OpenAI) | Actions are user-initiated; "robots.txt rules may not apply" | May not apply |
| Perplexity-User | "this fetcher generally ignores robots.txt rules" | Generally ignores |
| Claude-User (Anthropic) | Disabling Claude-User prevents retrieval of your content for user queries | Follows (per Anthropic) |
| Google user-triggered fetchers | "generally ignore robots.txt rules" | Generally ignore |

From OpenAI's docs: "ChatGPT-User is not used for crawling the web in an automatic fashion", and it is not used to decide whether content appears in Search. Perplexity writes that Perplexity-User is not used for web crawling or to collect training data.

Anthropic's docs differ: they say its bots follow robots.txt directives and support Crawl-delay. So the companies do not behave alike. Read each one's page separately; these docs can change.

## What does "ignores" mean, and what does it not mean?

- The bot visits your page like a normal visitor, only when a user supplies that address.
- It does not mean the bot crawls your whole site or gathers training data. The companies' docs say otherwise.
- A bot that does not know the address cannot reach the page. The real risk is the address reaching a user.

## How do you protect a private page?

Google's robots.txt guide says robots.txt is not a mechanism for keeping a page out of Google; to keep a page out, block indexing with noindex or password-protect it. The same logic holds for AI bots.

| Goal | Method | Note |
| --- | --- | --- |
| Nobody should see the page | Login (password, session) | The only real protection. A bot that cannot log in cannot see the content |
| Keep it out of search | `noindex` tag | The page stays public; anyone with the address can open it |
| Cut bot traffic | Firewall rule | Match the bot's user-agent and published IP range; enforceable |
| Tell bots your preference | robots.txt | Works for bots that comply; not protection |

If you cut bots at the firewall, do not rely on the user-agent alone; the text can be faked. OpenAI, Perplexity and Anthropic publish IP lists as JSON (for example `openai.com/chatgpt-user.json` for ChatGPT-User and `perplexity-user.json` for Perplexity-User). Anthropic does not recommend IP blocking as a reliable opt-out; it says such methods "may not work correctly".

A trade-off: if you block a user-triggered bot, the person using it cannot have your page used in their answer. Anthropic also says blocking can reduce visibility in user-directed search. For pages that are not private, a block is usually unnecessary.

## If you use Cloudflare

In Cloudflare, AI bots can fall under the "Agent" category, which covers bots acting in real time on a person's behalf. For settings and conflicts, see our [Cloudflare guide](https://ruesandora.com/writing/is-cdn-blocking-ai-bots).

## What does Ruescan show here?

Ruescan shows, for each AI bot, whether robots.txt allows it, blocks it or does not mention it. That shows your stated preference; it does not show whether user-triggered bots follow it. For a private page, use one of the methods above. No sign-up.

## Frequently asked questions

### What happens if I block ChatGPT-User in robots.txt?

Per OpenAI, the rule may not apply. The docs also say this bot does not decide Search inclusion. For a truly private page, use a login instead.

### Should I list a private page in robots.txt?

No. robots.txt is public; it shows everyone the address you disallowed. Keep private addresses out of it.

### Does noindex stop AI bots too?

Not verified. noindex is defined for Google; we found no statement in the AI companies' docs that their bots follow it.

### Does Perplexity-User collect training data?

Perplexity's docs say it is not used for web crawling or to collect training data. We could not verify this independently.

## Sources

Read on 16 October 2026.

- OpenAI, crawler documentation: [developers.openai.com/api/docs/bots](https://developers.openai.com/api/docs/bots)
- Perplexity, crawler documentation: [docs.perplexity.ai](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)
- Anthropic, web crawling and blocking the bot: [support.claude.com](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- Google Search Central, user-triggered fetchers: [developers.google.com](https://developers.google.com/search/docs/crawling-indexing/google-user-triggered-fetchers)
- Google Search Central, introduction to robots.txt: [developers.google.com](https://developers.google.com/search/docs/crawling-indexing/robots/intro)
- Cloudflare, Block AI bots: [developers.cloudflare.com](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/)

## Want to talk about this for your brand?

Write about your project and what you want to do. The message goes straight to Rues.

[Open the contact form](https://ruesandora.com/#iletisim)

[Email info@ruesandora.com](mailto:info@ruesandora.com) [Telegram ruesandora](https://t.me/ruesandora) [X @ruesandora0](https://x.com/ruesandora0) [Instagram @ruesandora0](https://www.instagram.com/ruesandora0) [YouTube @ruesandora0](https://www.youtube.com/@ruesandora0)

Page: https://ruesandora.com/writing/user-triggered-ai-bots