[Writing](https://ruesandora.com/writing/) Guide

# What is Content-Signal? Three AI permissions in robots.txt

By [Rues](https://ruesandora.com/about) · Published 23 October 2026 · Updated 23 October 2026

Sources checked on 20 October 2026

Short answer

Content-Signal is a line added to robots.txt that says yes or no, separately, to using your content for search, as input to AI answers, and for model training. Cloudflare announced it in September 2025. It is not an IETF standard, bots follow it voluntarily, and it does not block access; it only states your preference.

Content-Signal is a single line you add to your robots.txt file. It states whether your content may be used for three separate purposes: search, input to AI answers, and AI model training. It is a statement of preference; it does not technically stop any bot.

## What does the Content-Signal line do?

A classic robots.txt tells a bot "you may enter this address, not that one." Content-Signal says what the bot may do with the content after it has accessed it. Cloudflare explains the gap this way: robots.txt governs access, but does not say how content may be used once accessed.

An example robots.txt:

```
User-Agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
```

Here the site owner allows search and AI-answer input and does not allow model training.

## What do the three signals mean?

The definitions from the policy text Cloudflare published:

| Signal | What it means |
| --- | --- |
| `search` | Building a search index and providing search results (for example, returning links and short excerpts). It does not include AI-generated search summaries. |
| `ai-input` | Inputting content into one or more AI models (for example retrieval-augmented generation, grounding, or other real-time taking of content for generative AI search answers). |
| `ai-train` | Training or fine-tuning AI models. |

The same text says how to read the values: `yes` means you may collect the content for that use; `no` means you may not. If you leave a signal out, the site owner neither grants nor restricts permission for that use via the signal. So leaving it out does not mean "no preference"; it means "not said on this line".

The policy text also says that any restriction expressed this way is an express reservation of rights under Article 4 of the European Union's Copyright Directive 2019/790. We have not verified whether that has any legal counterpart in Turkey. This is not legal advice.

## Who uses Content-Signal, and what is the default?

Cloudflare announced the policy on 24 September 2025. It wrote that customers had by then turned on its managed robots.txt feature for over 3.8 million domains. As of the September 2025 announcement, the default was to add `search=yes, ai-train=no` to the robots.txt Cloudflare serves for those domains (the post says it "will update" the file). It did not add an `ai-input` signal, saying it did not know customers' preference and did not want to guess.

The default may have changed since: the managed robots.txt example in Cloudflare's documentation of 8 October 2026 reads `Content-signal: search=yes, ai-train=no, use=reference`, so `use=reference` is added too. The same documentation adds two things. On Free-plan domains that have no robots.txt of their own and do not use the managed feature, a bot requesting robots.txt sees only the policy text; that text expresses no preference. To set a preference, you have to configure your AI bot policies and turn on Bot Preference Sync. Cloudflare is also testing a fourth, optional field called `content-use`, with the values `use=immediate`, `use=reference` and `use=full`. The page says this is in testing.

If you do not use Cloudflare, you can write the line yourself. Cloudflare released the policy under a CC0 licence for anyone to use.

## Is Content-Signal a standard?

Not yet. The format of this line is Cloudflare's proposal. There is a separate standards effort, and its syntax is different:

- robots.txt itself is a standard, RFC 9309 (September 2022). The RFC says crawlers MAY interpret other records that are not part of the protocol. So reading a line like Content-Signal is optional.
- The IETF's AI Preferences (aipref) working group is working on two drafts: a vocabulary (draft-ietf-aipref-vocab, version 8, 14 September 2026) and an attachment format (draft-ietf-aipref-attach, version 5, August 2026). The attachment draft defines a `Content-Usage` rule for robots.txt, for example `Content-Usage: train-ai=n`. The names in the vocabulary differ too (`train-ai`, `search`).
- Both are working documents, not adopted standards. The vocabulary draft's own note says its contents do not reflect working group consensus.

So today there are two separate notations: Cloudflare's widely seen `Content-Signal` line and the still-draft IETF `Content-Usage` rule. We do not know which will last.

## Do bots follow Content-Signal?

Not verified. We found no document from a major AI or search company saying it will honour this line. What we do know:

- Cloudflare's own documentation says robots.txt compliance is voluntary and some crawlers may ignore it.
- Google's robots.txt documentation (last updated 31 August 2026) lists the fields it supports as `user-agent`, `allow`, `disallow` and `sitemap`. Content-Signal is not on that list.
- Cloudflare writes that Google Search Console may occasionally report "Syntax not understood" for this line, but that it has observed no impact on crawling rates or SEO. That is Cloudflare's own observation; we did not verify it.

So read the line as an open statement, not as a working barrier.

## What if I want to block access?

Do not mistake Content-Signal for a blocking tool. There are three separate layers:

| Layer | What it does | How binding |
| --- | --- | --- |
| Content-Signal | States a usage preference | Voluntary compliance |
| A bot-specific `Disallow` in robots.txt | Asks a particular bot to stay out | Voluntary compliance; OpenAI, Anthropic and Perplexity document this route for their own bots |
| A firewall or CDN rule | Cuts the request at the door | Enforced technically |

Our article on which bot to allow and which to close: [Robots.txt for AI bots](https://ruesandora.com/writing/robots-txt-for-ai-bots). And on a CDN cutting a bot that robots.txt allows: [Is Cloudflare blocking AI bots you allowed in robots.txt?](https://ruesandora.com/writing/is-cdn-blocking-ai-bots)

## Should I add it to my site?

That is your decision. No company's documentation tells you to add it, and none tells you not to. A balanced summary of what we know:

- Adding it: it puts your preference in writing. If bots that honour it appear, or in a dispute, you would have a record of your stated preference (we did not verify its legal value).
- Not adding it: no major bot is verified to follow it today, and there is no known risk either. Google Search Console may show a warning.
- Which value to choose is a business decision: `search=yes` if you want to appear in search; `yes` or `no` for AI-answer input and training is your call.

Ruescan is Rueseo's site scanner. One of its 275 signals checks whether robots.txt has a Content-Signal line. It is an information note and does not change the score, because it is not clear that major bots follow it ([signal glossary](https://ruesandora.com/glossary#ai-14)).

## Frequently asked questions

### Does Content-Signal replace Disallow in robots.txt?

No. They say different things: Disallow is about access, Content-Signal is about use. Both rely on voluntary compliance.

### What is the difference between `ai-input` and `ai-train`?

`ai-input` is taking content at the moment an answer is produced (grounding, for example). `ai-train` is training or fine-tuning a model. Those are Cloudflare's definitions; a bot says for itself which one it is doing, and you may not be able to verify that from outside.

### I do not use Cloudflare. Can I still write it?

Yes. The line is plain text and can go in any robots.txt. Cloudflare writes that it released the policy under CC0 for anyone to use.

### Is writing only `ai-train=no` enough?

As a format, yes; for the signals you leave out, you state no preference. As of the September 2025 announcement, Cloudflare's managed robots.txt wrote `search=yes, ai-train=no` in just this way; the example in its documentation of 8 October 2026 adds `use=reference`.

### Do I need `search=yes` to appear in Google's AI summaries?

No. Google's AI features documentation says there are no technical requirements beyond the page being indexed and eligible to be shown in Search with a snippet. Google's robots.txt documentation does not list Content-Signal among its supported fields either.

## Sources

All read on 20 October 2026.

- Cloudflare Blog, Giving users choice with Cloudflare's new Content Signals Policy (24 September 2025): [blog.cloudflare.com](https://blog.cloudflare.com/content-signals-policy/)
- Cloudflare, Managed robots.txt, Content Signals Policy section (last updated 8 October 2026): [developers.cloudflare.com](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/)
- Content Signals, Cloudflare's policy site: [contentsignals.org](https://contentsignals.org/)
- IETF, A Vocabulary For Expressing AI Usage Preferences, draft-ietf-aipref-vocab-08 (14 September 2026, working draft): [datatracker.ietf.org](https://datatracker.ietf.org/doc/draft-ietf-aipref-vocab/)
- IETF, draft-ietf-aipref-attach-05 (August 2026, working draft): [datatracker.ietf.org](https://datatracker.ietf.org/doc/draft-ietf-aipref-attach/)
- RFC 9309, Robots Exclusion Protocol (September 2022), section 2.2.4: [rfc-editor.org](https://www.rfc-editor.org/rfc/rfc9309.txt)
- Google Search Central, robots.txt documentation (last updated 31 August 2026): [developers.google.com](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt)
- Google Search Central, AI features and your website (last updated 10 December 2025): [developers.google.com](https://developers.google.com/search/docs/appearance/ai-features)

## Want to talk about this for your brand?

Write about your project and what you want to do. The message goes straight to Rues.

[Open the contact form](https://ruesandora.com/#iletisim)

[Email info@ruesandora.com](mailto:info@ruesandora.com) [Telegram ruesandora](https://t.me/ruesandora) [X @ruesandora0](https://x.com/ruesandora0) [Instagram @ruesandora0](https://www.instagram.com/ruesandora0) [YouTube @ruesandora0](https://www.youtube.com/@ruesandora0)

Page: https://ruesandora.com/writing/what-is-content-signal