What is Content-Signal? Three AI permissions in robots.txt

Content-Signal is a single line you add to your robots.txt file. It states whether your content may be used for three separate purposes: search, input to AI answers, and AI model training. It is a statement of preference; it does not technically stop any bot.

What does the Content-Signal line do?

A classic robots.txt tells a bot "you may enter this address, not that one." Content-Signal says what the bot may do with the content after it has accessed it. Cloudflare explains the gap this way: robots.txt governs access, but does not say how content may be used once accessed.

An example robots.txt:

User-Agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

Here the site owner allows search and AI-answer input and does not allow model training.

What do the three signals mean?

The definitions from the policy text Cloudflare published:

The same text says how to read the values: yes means you may collect the content for that use; no means you may not. If you leave a signal out, the site owner neither grants nor restricts permission for that use via the signal. So leaving it out does not mean "no preference"; it means "not said on this line".

The policy text also says that any restriction expressed this way is an express reservation of rights under Article 4 of the European Union's Copyright Directive 2019/790. We have not verified whether that has any legal counterpart in Turkey. This is not legal advice.

Who uses Content-Signal, and what is the default?

Cloudflare announced the policy on 24 September 2025. It wrote that customers had by then turned on its managed robots.txt feature for over 3.8 million domains. As of the September 2025 announcement, the default was to add search=yes, ai-train=no to the robots.txt Cloudflare serves for those domains (the post says it "will update" the file). It did not add an ai-input signal, saying it did not know customers' preference and did not want to guess.

The default may have changed since: the managed robots.txt example in Cloudflare's documentation of 8 October 2026 reads Content-signal: search=yes, ai-train=no, use=reference, so use=reference is added too. The same documentation adds two things. On Free-plan domains that have no robots.txt of their own and do not use the managed feature, a bot requesting robots.txt sees only the policy text; that text expresses no preference. To set a preference, you have to configure your AI bot policies and turn on Bot Preference Sync. Cloudflare is also testing a fourth, optional field called content-use, with the values use=immediate, use=reference and use=full. The page says this is in testing.

If you do not use Cloudflare, you can write the line yourself. Cloudflare released the policy under a CC0 licence for anyone to use.

Is Content-Signal a standard?

Not yet. The format of this line is Cloudflare's proposal. There is a separate standards effort, and its syntax is different:

• robots.txt itself is a standard, RFC 9309 (September 2022). The RFC says crawlers MAY interpret other records that are not part of the protocol. So reading a line like Content-Signal is optional.
• The IETF's AI Preferences (aipref) working group is working on two drafts: a vocabulary (draft-ietf-aipref-vocab, version 8, 14 September 2026) and an attachment format (draft-ietf-aipref-attach, version 5, August 2026). The attachment draft defines a Content-Usage rule for robots.txt, for example Content-Usage: train-ai=n. The names in the vocabulary differ too (train-ai, search).
• Both are working documents, not adopted standards. The vocabulary draft's own note says its contents do not reflect working group consensus.

So today there are two separate notations: Cloudflare's widely seen Content-Signal line and the still-draft IETF Content-Usage rule. We do not know which will last.

Do bots follow Content-Signal?

Not verified. We found no document from a major AI or search company saying it will honour this line. What we do know:

• Cloudflare's own documentation says robots.txt compliance is voluntary and some crawlers may ignore it.
• Google's robots.txt documentation (last updated 31 August 2026) lists the fields it supports as user-agent, allow, disallow and sitemap. Content-Signal is not on that list.
• Cloudflare writes that Google Search Console may occasionally report "Syntax not understood" for this line, but that it has observed no impact on crawling rates or SEO. That is Cloudflare's own observation; we did not verify it.

So read the line as an open statement, not as a working barrier.

What if I want to block access?

Do not mistake Content-Signal for a blocking tool. There are three separate layers:

Our article on which bot to allow and which to close: Robots.txt for AI bots. And on a CDN cutting a bot that robots.txt allows: Is Cloudflare blocking AI bots you allowed in robots.txt?

Should I add it to my site?

That is your decision. No company's documentation tells you to add it, and none tells you not to. A balanced summary of what we know:

• Adding it: it puts your preference in writing. If bots that honour it appear, or in a dispute, you would have a record of your stated preference (we did not verify its legal value).
• Not adding it: no major bot is verified to follow it today, and there is no known risk either. Google Search Console may show a warning.
• Which value to choose is a business decision: search=yes if you want to appear in search; yes or no for AI-answer input and training is your call.

Ruescan is Rueseo's site scanner. One of its 275 signals checks whether robots.txt has a Content-Signal line. It is an information note and does not change the score, because it is not clear that major bots follow it (signal glossary).

Frequently asked questions