If ChatGPT-User ignores robots.txt, how to protect a page?

A user tells ChatGPT "read this address", and ChatGPT-User fetches your page, even if you disallowed it in robots.txt. This guide covers what each company says and how to protect a page that really must stay private.

What is a user-triggered bot?

AI company bots do one of three jobs: index for search, collect for model training, or fetch a page at that moment to answer a user's question. The third kind is a user-triggered bot. It does not roam on its own; it runs because a person asked.

What does each company say?

From OpenAI's docs: "ChatGPT-User is not used for crawling the web in an automatic fashion", and it is not used to decide whether content appears in Search. Perplexity writes that Perplexity-User is not used for web crawling or to collect training data.

Anthropic's docs differ: they say its bots follow robots.txt directives and support Crawl-delay. So the companies do not behave alike. Read each one's page separately; these docs can change.

What does "ignores" mean, and what does it not mean?

• The bot visits your page like a normal visitor, only when a user supplies that address.
• It does not mean the bot crawls your whole site or gathers training data. The companies' docs say otherwise.
• A bot that does not know the address cannot reach the page. The real risk is the address reaching a user.

How do you protect a private page?

Google's robots.txt guide says robots.txt is not a mechanism for keeping a page out of Google; to keep a page out, block indexing with noindex or password-protect it. The same logic holds for AI bots.

If you cut bots at the firewall, do not rely on the user-agent alone; the text can be faked. OpenAI, Perplexity and Anthropic publish IP lists as JSON (for example openai.com/chatgpt-user.json for ChatGPT-User and perplexity-user.json for Perplexity-User). Anthropic does not recommend IP blocking as a reliable opt-out; it says such methods "may not work correctly".

A trade-off: if you block a user-triggered bot, the person using it cannot have your page used in their answer. Anthropic also says blocking can reduce visibility in user-directed search. For pages that are not private, a block is usually unnecessary.

If you use Cloudflare

In Cloudflare, AI bots can fall under the "Agent" category, which covers bots acting in real time on a person's behalf. For settings and conflicts, see our Cloudflare guide.

What does Ruescan show here?

Ruescan shows, for each AI bot, whether robots.txt allows it, blocks it or does not mention it. That shows your stated preference; it does not show whether user-triggered bots follow it. For a private page, use one of the methods above. No sign-up.

Frequently asked questions