Officially, the major AI engines say their crawlers respect robots.txt, and each publishes per-bot tokens that let site owners opt out of training and/or search. A controlled experiment reported on r/SEO in September 2026 found a different picture in the logs: at least some AI crawler traffic ignored robots.txt entirely and arrived as Chrome and Safari browser requests. The honest synthesis of the two sides: robots.txt is a permission signal and a courtesy protocol, not a security control — it stops the bots that tell you their name, and if your content must not be quoted, the real control belongs at the content and technical layers, not in the file alone.
What did the September 2026 practitioner experiment find?
One engine was reported to respect robots.txt, one partially, and the rest of the traffic behaved like browsers, not bots. A practitioner (u/hunxai69) ran a controlled experiment in September 2026: a set of secret, unlinked test URLs, one per AI engine, plus server-side logging of IP addresses and user agents. That setup isolates crawl traffic from human visitors and shows exactly which requests read robots.txt before touching the test pages.
As reported in the experiment:
- Claude respected robots.txt on new sites.
- ChatGPT’s first hit ignored robots.txt, after which OAI-SearchBot read the file and its behavior changed.
- Gemini requests used a bare “Google” user agent from IP addresses not on Google’s published crawler ranges, and never read robots.txt.
- Grok hit the test URLs from 59 IPs across 6 requests, all masquerading as Chrome or Safari from proxy networks (including Oxylabs), with zero robots.txt reads.
The practitioner’s own conclusion, as reported: “Blocking by user agent clearly only stops the bots that tell you their name.”
One verification caveat, stated plainly: the u/hunxai69 account exists on Reddit (about three years old) but has hidden all posts, so the original thread was not publicly retrievable when we checked. Cite it as it should be cited — a controlled experiment reported on r/SEO (September 2026, u/hunxai69), the original thread not publicly retrievable at verification time (checked 2026-10-29). The specific figures above (59 IPs, the Oxylabs attribution, the bare “Google” user agent) appear only in that report. We label them as reported, not independently confirmed — and for Gemini specifically, Google’s own documentation offers an explanation that makes log-based attribution genuinely ambiguous, which we cover below.
Is this one person’s log, or a pattern?
The stealth-crawling pattern appears in at least two independent, verifiable reports from different organizations and different dates. Neither is a large-scale study, but neither is one person’s log:
- DataDome, via PPC Land (Luis Rijo, January 6, 2026). On December 6, 2025, DataDome VP of Threat Research Jerome Segura asked Grok to fetch a single page. The request generated 16 requests from 12 IPs with spoofed user agents — Chrome on macOS, Safari on iPhone, and Go-http-client — reported in DataDome’s analysis of AI agents masquerading as humans.
- Crawler-Canary-Lab Part 3 (GitHub, servodecristorj-ai, August 19–26, 2026). A reproducible canary-token study (public repo) found Grok running in dual mode: an honest mode with xAI-Bot / Grok-Bot user agents, and a stealth mode using a Safari/605.1.15 user agent from 3 ASNs including Web2Objects. The study’s explicit conclusion: robots.txt-only blocking is insufficient.
So the strongest defensible statement is: multiple independent reports from late 2025 through summer 2026 describe the same engine (Grok) running parallel stealth traffic that spoofs browser user agents and never reads robots.txt. Whether other engines do the same, at what volume, and whether it is deliberate policy or a misconfigured secondary fetcher, is not established by any published data we could verify.
What do the official crawler docs actually promise?
Every major engine documents a robots.txt mechanism, and Google goes furthest on verification. The official positions, checked against the primary documentation on 2026-10-29:
- Google. Introduction to robots.txt (updated 2025-12-10): “A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google. To keep a web page out of Google, block indexing with noindex or password-protect the page.” The list of Google’s common crawlers adds that they “always obey robots.txt rules when crawling automatically,” that they generally crawl from the IP ranges published in the common-crawlers.json object with matching reverse DNS, and — the important caveat — “The HTTP user agent string can be spoofed.”
- Google’s verification path. Verify requests from Google crawlers and fetchers describes the official check: reverse DNS lookup, the resulting domain must be googlebot.com, google.com or googleusercontent.com, and the forward DNS must match the original IP — or match against Google’s published IP lists (common-crawlers.json, special-case-crawlers-google.json, user-triggered-agents.json). An experimental “Authenticate requests with Web Bot Auth” mechanism is also documented.
- Google-Extended, the Gemini training token. The key passage: “Google-Extended doesn’t have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity.” In other words, Google says Gemini training crawls arrive under ordinary Google user-agent strings, with the Google-Extended token in robots.txt deciding whether crawled content may train future Gemini models (and be used for grounding), while “not impact[ing] a site’s inclusion in Google Search nor…used as a ranking signal.”
- OpenAI. Overview of OpenAI Crawlers: “OpenAI uses OAI-SearchBot and GPTBot robots.txt tags to enable webmasters to manage how their sites and content work with AI. Each setting is independent of the others – for example, a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training OpenAI’s generative AI foundation models.” OpenAI also says it can take “~24 hours” from a robots.txt update for its systems to adjust, and that “ChatGPT-User is not used for crawling the web in an automatic fashion. Because these actions are initiated by a user, robots.txt rules may not apply.”
- Anthropic. Does Anthropic crawl data from the web, and how can site owners block the crawler? is the official help article, covering three roles: ClaudeBot (training data collection), Claude-SearchBot (search quality and indexing), and Claude-User (user-directed fetches).
- xAI. No official robots.txt policy page was found on x.ai at verification time (2026-10-29). We are not claiming xAI publishes an official crawler policy — the absence is itself the finding: for one of the four engines in the experiment, there is no documented promise to hold it to.
| Engine | Official robots.txt mechanism (verified 2026-10-29) | What practitioners reported |
|---|---|---|
| Google / Gemini | Google-Extended token in robots.txt; crawls run under existing Google user-agent strings; claims crawlers “always obey” robots.txt; official reverse-DNS + published-IP verification | Single report (Sept 2026, u/hunxai69): bare “Google” UA from IPs off Google’s published ranges, no robots.txt reads — ambiguous given the documented mechanism |
| OpenAI / ChatGPT | Two independent tokens: OAI-SearchBot (search) and GPTBot (training); ~24h to adjust; ChatGPT-User exempt from robots.txt | Reported (Sept 2026): first hit ignored robots.txt, then OAI-SearchBot read it and behavior changed |
| Anthropic / Claude | Official help article documents ClaudeBot (training), Claude-SearchBot (search), Claude-User (user-directed) | Reported (Sept 2026): respected robots.txt on new sites |
| xAI / Grok | No official robots.txt policy page found on x.ai | Multiple reports: r/SEO experiment (59 IPs / 6 requests, browser UAs, zero robots.txt reads), DataDome (16 requests / 12 IPs, spoofed UAs, Dec 2025), Crawler-Canary-Lab (dual honest/stealth mode, Aug 2026) |
Does robots.txt actually stop AI crawlers?
It stops the bots that identify themselves — and that is exactly what Google’s own documentation says it is and is not. Read the two Google passages together: robots.txt “is not a mechanism for keeping a web page out of Google,” and “the HTTP user agent string can be spoofed.” That is the entire honest position in two sentences. robots.txt is a client-side permission signal: the crawler fetches it, and the crawler decides. There is no handshake, no enforcement, no audit trail. Every one of the practitioner findings above is consistent with that framing, because none of it requires a crawler to be caught — a stealth fetcher that spoofs a browser user agent simply never presents a token for your rules to match.
That last point matters for how you read the Gemini finding. Because Google documents that Gemini training crawls run under existing Google user-agent strings, a “Google”-labeled request from an odd IP range is not automatically proof of noncompliance — it could be the documented mechanism seen through a log that didn’t run the reverse-DNS verification. We are not claiming Gemini’s crawls “never read robots.txt”; that is one practitioner’s report, and the docs leave an alternative reading. What we can say, from the verified corroborations, is that Grok’s stealth mode is documented in at least two independent reports with the same signature: browser user agents, distributed IPs, no robots.txt fetch.
Two more boundaries worth stating. First, user-directed fetches (ChatGPT-User, Claude-User) are documented by their vendors as exempt from or not subject to robots.txt — a user asking the engine to read your page is not something you can reliably block, and usually shouldn’t try. Second, robots.txt is one layer of a stack; the crawl mechanics — training vs grounding roles, the plain HTTP fetch path, what a JavaScript shell returns — are worked through in how AI engines crawl your site and what breaks silently, and the contractual layer (what your hosting terms say about AI crawling, independent of any file) is covered in Meta’s website-crawling clause.
How do you actually control what AI engines can quote?
Treat robots.txt exclusions as opt-outs of known, well-behaved crawlers — not as walls. If a page must not be quoted (pre-launch pricing, confidential data, terms you don’t want recombined into answers), the control is at the content layer or the technical layer. A practical checklist, in order:
- Set explicit per-bot rules for every engine you care about. Name GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot and Google-Extended individually, with a deliberate allow or disallow for each. Silence is not a policy — a wildcard
Disallow: /blocks everything you never thought to name, and a missing rule means default-allowed. - Audit your server logs for unattributed crawler bursts. Look for bursts from many IPs sharing browser user agents that never touch robots.txt first — the exact signature in the Grok reports. If you see it, you have your own local version of the experiment.
- Verify Google requests if spoofing is suspected. Run Google’s official check — reverse DNS to a googlebot.com/google.com/googleusercontent.com domain with a matching forward DNS, or a match against the published IP lists — before you trust (or accuse) any “Google” hit.
- Decide per page what should be quotable. robots.txt manages crawling; it does not manage why an answer engine would pick your sentence. The content-level work — clear claims, first-hand detail, unambiguous phrasing — is what actually decides whether you get quoted, and it is where the durable control lives. The mechanics are in what makes content AI-citable.
- Put real secrets in real controls. Auth walls, non-public URLs, and simply not publishing sensitive pages are the layers that work. If content is publicly indexable, assume a sufficiently determined fetcher — bot or not — can reach it, and plan the sensitivity of what you publish accordingly.
And keep the promise in check: nothing in the docs or the experiments says blocking a crawler reliably keeps your content out of AI answers. Blocked text may still leak via search snippets, user-directed fetches, caches or the parallel traffic described above. Blocking is an opt-out of known crawlers; confidentiality is a publishing decision.
Where to go from here
If you want this run against your actual logs — which crawlers are hitting you, which are spoofing, what your robots.txt is silently allowing, and what should be quotable per page — that is a defined audit in our methodology, and it is the fastest way to turn the permission file from an afterthought into a policy you can defend.
Sources checked against the linked originals on 2026-10-29.