Cloudflare's Disallow AI Training: stay indexed for search, refuse training — and audit your settings after the September 15 migration

On September 15, 2026, Cloudflare quietly rebuilt one of the most consequential settings a website owner will ever toggle. The old “Block AI Bots” switch — a single yes/no that blocked some AI crawlers while leaving mixed-use ones alone — was deprecated and replaced with three separate controls: Search, Training, and Agent, plus a new Disallow AI Training setting that publishes a no-training preference to robots.txt while keeping you indexed for search. The catch that most write-ups miss: your old settings were migrated automatically under you, and the new “Block” now means something different — it can stop Googlebot and Bingbot, search included. This post covers what changed, the verified migration table, the decision it forces on a B2B SaaS site, and the ten-minute audit you should run whether or not you care about AI training.

Everything below is checked against Cloudflare’s own announcement (Bryan Becker, September 15, 2026: https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) and a settings walkthrough published October 5, 2026 (TechSansar: https://techsansar.com/computing/cloudflare-ai-visibility-settings/), which notes it plainly: “If you ever enabled Block AI Bots, your settings changed under you.”

What changed on September 15

Cloudflare’s framing: the hard part of the AI training question was the mixed-use crawler — a single crawler that serves both search indexing and AI training (Googlebot, Bingbot and Applebot all qualify). Under the old controls you could not refuse one use without refusing the other, so site owners were pushed into an either/or: allow training, or risk discoverability.

The rebuild splits that:

• Three behaviors, three controls. Search (crawling to build a search index), Training (crawling to train or fine-tune a model), and Agent (user-directed agents fetching pages on a human’s behalf — chat fetch bots, browser-use agents). Each has its own Allow / Block / Block-on-pages-with-ads value; Training additionally gets Disallow AI Training.
• Disallow AI Training works by publishing the applicable no-training preference in robots.txt (via Bot Preference Sync — the successor to Managed Robots.txt). Accountable mixed-use crawlers — Cloudflare’s designation for Apple, Google and Microsoft, who honor or have committed to honoring the preference — keep crawling for search. Every other training crawler is blocked at the edge, “including the training-only crawlers run by Amazon, Anthropic, Meta, and OpenAI — blocking those does not affect search.”
• The Accountable designation. To qualify, a bot operator must meet or commit to: an opt-out mechanism for AI training (robots.txt or similar); an opt-out for AI summaries; URL-level visibility into what was made available for training; and assurance that opting out of training does not affect traditional search.
• “Block” now includes search bots. Block and “Block on pages with ads” now apply to mixed-use crawlers — Applebot, Bingbot, Googlebot — “so either setting impacts search as well as training.”
• No Disallow for agents, yet. There is no Disallow setting for the Agent class — Cloudflare says it will revisit once standards such as ai-prefs mature.

One stat from the post that summarizes the market’s actual preference: less than 1% of Cloudflare sites block Search bots, while 17% enable some mechanism to block training. The default posture of the installed base is: be found, refuse to be trained on.

The migration table (verify your row)

Cloudflare migrated existing domains automatically. For sites that never configured the granular controls, the mapping from the legacy “Block AI Bots” setting is:

For domains that had configured the granular Search/Training/Agent controls, Cloudflare says it preserved the practical effect of the selections — previous Training values of “Block” or “Block on pages with ads” migrated to Disallow AI Training. Its summary: “Nothing, in almost every case. Your current settings carry over on their own.” Note what that means in practice: if you had legacy “Block” on, you are now explicitly disallowing AI training on every training crawler except the accountable mixed-use ones — a real policy position you may never have consciously chosen.

The audit note (10 minutes, do it now)

Because the migration happened silently and the semantics changed, run this regardless of your AI strategy:

1. Open the new control. Dashboard → your domain → Security → Settings → Configure AI bot policies. Confirm the three values. For any site whose revenue depends on being found in Google or Bing, Search must be Allow — and under the new taxonomy, a stray “Block” there stops Googlebot and Bingbot, not just AI bots.
2. Decide the Training row consciously. Allow (status quo, train freely), Disallow AI Training (the setting most B2B SaaS sites want — more below), or the ad-page variants. If the migrated value surprises you, change it; the change is a single toggle plus a robots.txt update that Bot Preference Sync handles.
3. Check what robots.txt now contains. With Bot Preference Sync, your robots.txt should reflect your training preference for tokens like Google-Extended and Applebot-Extended. One asymmetry to know: Bing’s robots.txt-level no-training mechanism is “targeted for early 2027” — Bingbot currently opts out via Microsoft’s NOARCHIVE meta tag, with granular controls in Bing Webmaster Tools (per Cloudflare’s post).
4. Confirm Googlebot is actually still arriving. Dashboard settings are intentions; the crawl stats are the fact. Search Console’s crawl stats, or your own access logs, should show Googlebot requests after September 15. If they stopped, work backward from here — the TechSansar walkthrough makes exactly this verification step part of any post-migration audit.

The decision, for a B2B SaaS site

Set aside the training ethics and look at the commercial structure. The Cloudflare data above is the honest prior: the installed base of sites overwhelmingly wants both — discoverability for search and refusal of training — and September 15 is the first time a CDN made that a one-click, enforced configuration rather than a robots.txt hope.

For a B2B SaaS site, the default we would recommend is Search: Allow, Training: Disallow AI Training, Agent: Allow — with the caveats that make it a decision rather than a reflex:

• What Disallow AI Training does: stops training crawlers (accountable mixed-use ones for search, blocked at the edge for everyone else’s training-only bots), publishes the preference publicly in robots.txt, and — per Apple and Google as relayed in Cloudflare’s post — does not affect search ranking.
• What it does not do: it does not control the Agent class (no Disallow exists yet), and it does not control AI summaries — per the announcement, a per-operator summary opt-out is a requirement of the Accountable designation, and Cloudflare’s goal is a single site-wide control “by early next year.” If content-in-summaries is the thing you actually care about, that control is not here yet.
• It is a policy, not a traffic lever. Nothing in the verified sources says blocking training costs you AI-answer citations — those come from the Search and Agent paths, which stay open. Treat it the same way you treat Google-Extended disallowance: a statement about your data, free of ranking consequences, and cheap to reverse.
• If you are not on Cloudflare, the same decision exists in your own robots.txt and server config — the September 15 change mostly matters because it normalizes the vocabulary and enforces the preference at the edge for the largest slice of the web.

The crawler mechanics this sits on top of — which bot does what, what robots.txt can and cannot enforce — are the subject of how AI engines crawl your site, and if you are auditing the file itself, the llms.txt spec audit is the natural companion.

Next step: the toggle takes five minutes; knowing what the new settings do to your AI visibility over the following months is the ongoing part. That is what we track, and how — in our methodology.