Cloudflare’s Content Signals is a policy that adds three directives to robots.txt — search, ai-input, and ai-train — letting an operator state preferences for how their content may be used after it has been accessed. The “alternative to llms.txt” framing is wrong, and worth correcting: llms.txt is a content index for LLMs (what to read), while Content Signals is a permission signal in robots.txt (what you may do with it). They answer different questions for different systems and are complements, not substitutes. In practice, Content Signals has displaced llms.txt as the more consequential AI-facing signal — it rides on robots.txt, which every major crawler already reads; Cloudflare serves it at the edge for over 3.8 million domains already on managed robots.txt; and it carries a legal framing llms.txt never will: an express rights-reservation clause under Article 4 of the EU DSM Directive. None of this kills llms.txt — it survives as a courtesy index for inference-time agents (what llms.txt is).
What Content Signals actually is
Cloudflare announced the policy on September 24, 2025 (Will Allen: https://blog.cloudflare.com/content-signals-policy/), motivated by the free-rider problem — companies scraping at scale with no compensation, in a world where Cloudflare expects bot traffic to exceed human traffic by end of 2029. It defines three signals, with these exact definitions:
search— “building a search index and providing search results (e.g., returning hyperlinks and short excerpts from your website’s contents). Search does not include providing AI-generated search summaries.”ai-input— “inputting content into one or more AI models (e.g., retrieval augmented generation, grounding, or other real-time taking of content for generative AI search answers).”ai-train— “training or fine-tuning AI models.”
The interpretation rules (published as human-readable comments in the robots.txt text itself) are deliberately simple: yes means you may collect content for that use; no means you may not; absent means the operator neither grants nor restricts permission for that use.
The machine-readable syntax is a comma-delimited yes/no list under a User-Agent line:
User-Agent: *
Content-Signal: search=yes, ai-train=no
Allow: /
Every policy text ends with this clause:
“ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.”
That is the part llms.txt cannot offer: turning a crawler convention into an explicit, on-the-record assertion of copyright. The policy is released under the CC0 license, so anyone — not just Cloudflare customers — can implement it.
Two deployment details matter if you are on Cloudflare:
- For managed-robots.txt customers who choose disallow-training, Cloudflare serves
Content-Signal: search=yes, ai-train=no— and deliberately does not serve an ai-input signal, because “we don’t know their preference with respect to that signal, and we don’t want to guess.” - Free-plan zones without an existing robots.txt automatically get the commented, human-readable version — no directives, a no-op by design, toggleable off in the dashboard.
And the caveat, in Cloudflare’s own words: “Content signals express preferences; they are not technical countermeasures against scraping. Some companies might simply ignore them.” Cloudflare’s recommendation is to combine them with WAF rules and Bot Management.
Content Signals vs. llms.txt, side by side
| Content Signals | llms.txt | |
|---|---|---|
| What it is | Permission directives inside robots.txt | A markdown index file, usually at your root |
| Question answered | What may you do with my content? | What do I have, and where is it? |
| Primary audience | Search crawlers, AI search systems, model operators | Inference-time LLM agents that fetch it |
| Enforcement | Served at the edge by Cloudflare; the “Accountable” mixed-use crawlers honor or have committed to honoring no-training; other training crawlers get blocked at the edge | None — purely voluntary; agents that don’t read it are unaffected |
| Adoption | 3.8M+ domains on Cloudflare managed robots.txt | “Thousands of sites” per the spec page’s own claims |
| Legal framing | Article 4, EU DSM Directive 2019/790, rights-reservation clause | None in the spec |
The honest limits, straight from contentsignals.org: “Not all automated systems honor robots.txt files, and some may ignore these Content-Signal directives,” and “courts and regulators may conclude that robots.txt files do not impose enforceable legal obligations.” Nothing in Content Signals technically blocks a determined scraper — the edge enforcement that exists is Cloudflare’s, and it only reaches sites behind Cloudflare. What robots.txt can and cannot enforce at all is the subject of how AI engines crawl your site.
What happened next: the September 15, 2026 rebuild
On September 15, 2026, Cloudflare rebuilt its AI-crawler settings into three controls — Search, Training, and Agent — added Disallow AI Training, and moved robots.txt publishing to Bot Preference Sync, the successor to Managed robots.txt. The “Accountable” mixed-use crawlers — Apple, Google, and Microsoft — honor or have committed to honoring the no-training preference while keeping your site indexed for search; every other training crawler (the training-only bots run by Amazon, Anthropic, Meta, and OpenAI) is blocked at the edge, without affecting search. The stat that frames the market: less than 1% of Cloudflare sites block Search bots, while 17% enable some mechanism to block training. The migration table and ten-minute audit checklist are in Cloudflare’s Disallow AI Training, so we won’t repeat them here. The relevant point for this post: Content Signals went from a robots.txt convention to the published record behind a control plane that Cloudflare actually enforces at the edge.
How to deploy it
1. The contentsignals.org generator. contentsignals.org has a generator where you pick a policy and get ready-to-serve robots.txt text, plus a “Deploy to Cloudflare” button if your zone is there.
2. Copy-paste into your own robots.txt. The policy is CC0 — paste the same text into /robots.txt on any host. It works identically for everyone not behind Cloudflare; you forgo edge enforcement, which was never part of the policy anyway.
3. Per user-agent and per path. The syntax supports targeting, exactly as published by Cloudflare. Per user-agent:
User-Agent: googlebot
Content-Signal: ai-train=no, search=yes, ai-input=no
Allow: /
Per path (different signals for different sections of the site):
Content-Signal: /about ai-train=yes, search=yes, ai-input=yes
Allow: /about
Content-Signal: /blog/ ai-train=no, search=yes, ai-input=no
Allow: /blog/
Disallow: /dashboard/
4. The four default policies in the generator, from most to least restrictive: Disallow All (the generator warns this “may cause search engines like Google to exclude your site from search results”), Allow Search Only, Allow Search & AI Input (which does not grant training permission), and Allow Search, AI Input & AI Training.
The decision, for a B2B SaaS site
The 17%-versus-<1% split is your prior, and it maps directly onto the three signals:
search=yes— always. For a B2B SaaS site, being indexed in Google and Bing is not negotiable, and that is the only default the generator itself warns about. If a third party proposes a “Disallow All” style block, treat it the way you would treat a Googlebot disallow: it is a commercial decision, not a technical one.ai-input=yes— if you want to be cited, and most B2B sites should. This is the nuance most articles miss:searchexplicitly excludes “AI-generated search summaries.” By the policy’s own definition, RAG, grounding and real-time generative-search answers are all ai-input — not search. A site that setssearch=yes, ai-input=nois telling AI search systems they may index it for classic results but not ground an answer in it. If citation in AI answers is a goal, ai-input is the lever that points at it — and it is free to grant, since nothing in the verified sources says it carries training risk.ai-train=no— the common B2B choice. Scraping at scale without compensation is the free-rider problem the policy exists to address, and Article 4’s rights-reservation clause gives a no-training stance a written legal posture rather than a robots.txt hope. It is reversible, and per the September 15 rebuild the Accountable mixed-use crawlers honor or committed to honoring it without affecting search.
None of this is a ranking signal: nothing in Cloudflare’s verified sources says Content Signals improves or hurts Google rankings; the only enforcement claim is the narrow one above — no-training honored by designated crawlers, training-only bots blocked at the edge, search unaffected.
For the index layer that sits next to this, the llms.txt v2 audit is the companion check.
Next step: setting the three directives takes ten minutes; knowing whether they changed how AI systems behave toward your site is the ongoing measurement problem. That is what we track, and how — in our methodology.