AI answer engines crawl your site in two distinct roles: training crawlers, which fetch content that may feed model training, and grounding crawlers, which build the indexes that let your pages appear in live search answers. The two roles have different crawler identities, different robots.txt tokens, and different costs when you block them — and every one of them fetches over plain HTTP, which means what a crawler sees is what an ordinary HTTP client gets. The most common failure is silent: when a grounding crawler stops reaching your pages, no dashboard, no console and no report tells you.

Two roles, not one “AI crawler”

Most crawling advice treats “AI bots” as one species. The vendors document it differently. (The full row-by-row list, with the official doc behind each bot, is our maintained LLM crawlers reference — that post is the list; this one is what the roles mean.)

Training crawlers collect content that may feed model training. OpenAI’s docs say GPTBot “is used to crawl content that may be used in training” its foundation models; Anthropic’s docs say ClaudeBot collects web content that “could potentially contribute to” training. Google’s version is not a bot at all: Google-Extended is a robots.txt token that controls whether Googlebot-collected content may train future Gemini models — and Google states in writing that it “doesn’t affect a site’s inclusion in Search, nor do we use Google-Extended as a ranking signal.”

Grounding (search/answer) crawlers build an index so your site can appear in live answers: OAI-SearchBot for ChatGPT search, Claude-SearchBot for Claude search, PerplexityBot for Perplexity. Each vendor says in writing that this bot does not train models. This is the layer where your citation presence is decided, so it is the one to check first: if it can’t fetch you, nothing else on this site matters.

User-directed fetchers fire when a human asks a question or clicks something (ChatGPT-User, Claude-User, Perplexity-User). Two vendors state that robots.txt rules “may not apply” or are “generally ignored” for these — a user’s direct request is not something you can reliably block, and usually shouldn’t try.

The practical consequence: “block AI” is not one switch. Training and grounding are independent axes per vendor. You can block GPTBot and ClaudeBot and stay fully visible in ChatGPT and Claude search. You can allow training and still not appear in any answer. Confusing the axes is how sites silently opt themselves out of the one thing they actually wanted.

The fetch path: plain HTTP, and what the fetcher actually sees

These crawlers are ordinary HTTP clients. They request your URL with a user-agent string and read the response. Three things follow:

They get what a plain client gets. If your top page is a JavaScript shell — an SPA whose content is assembled client-side — a plain fetch can return almost no text. Google has long operated a JavaScript-rendering pipeline for Googlebot, and nobody should assume any AI surface does the same: none of the vendor crawler documentation we verify in the reference describes JavaScript rendering for grounding crawlers, so the working assumption is that the static HTML response is what gets read. If your page needs a browser to exist, plan on the assumption that your answer engines are reading an empty room.

Firewalls are the real gate, often above robots.txt. A WAF that blocks unknown user agents or non-listed IPs will drop a bot that your robots.txt allows — and that block, like the robot, leaves no trace in any of your marketing tools. Perplexity’s own documentation ships WAF configuration steps (Cloudflare, AWS) precisely because this is a common failure. The only place the truth lives is your server logs: the requests arrive with the exact user-agent strings in the reference’s table.

Changes are slow to bite. OpenAI says robots.txt changes take “about 24 hours” to take effect in ChatGPT search; Perplexity says “up to 24 hours.” If you edited the file this morning, judge the result next week, not this afternoon.

robots.txt: silence has a cost

robots.txt is a client-side file. The crawler reads it and decides; there is no handshake, no notification, no report. Concretely: there is no dashboard that shows you OAI-SearchBot, PerplexityBot or Claude-SearchBot has stopped fetching you because of a rule. Google Search Console reports on Googlebot indexing and on your opt-out of Google’s own generative features — it says nothing about third-party AI crawlers being blocked. The block simply manifests as absence in the answers, weeks later, attributed to “AI is just inconsistent.”

Three silent killers, in order of how often we see them:

  1. The wildcard rule. A blanket User-agent: * / Disallow: / (from staging, a plugin, a password-protected area) applies to every agent that isn’t named separately — including every grounding crawler. You have never written a rule naming OAI-SearchBot, and you have still blocked it. This is the single most common “we never opted out” case.
  2. Subdomains. Anthropic’s docs are explicit: the opt-out rule must be added “for every subdomain that you wish to opt out from.” Your docs.yourproduct.com robots.txt is a separate file from yourproduct.com’s.
  3. The WAF above the file. Covered above: the firewall can block what the file allows, with equal silence.

One nuance so that blocking isn’t treated as total silence: Perplexity’s help center says a blocked page “may still [have] the domain, headline, and a brief factual summary” indexed. A block is usually a heavy reduction, occasionally not a hard zero.

llms.txt: the index layer, not a ranking lever

If the crawlers are the legs, llms.txt is the map: a proposed convention (v2 spec at https://llmstxt.org/, modified August 10, 2026) in which a markdown file at your site root gives LLM agents a concise machine-readable index of your content, with links to per-page markdown versions. The spec makes no ranking or citation claim — it is a courtesy index for the agents that fetch it, and its role in the stack is to make the fetch path above cheap and reliable once a crawler is on your domain. The audit of what v2 actually requires is in llms.txt in 2026: what the v2 spec actually requires.

So the crawl stack in one line: crawlers must reach you (robots.txt, WAF, plain HTML) → the index layer tells agents where things are (llms.txt) → the content must be worth citing (the rest of the series).

The 5-minute self-check

Two commands plus a log glance. Replace yoursite.com and run them now:

# 1. What a plain client gets back from your top page (size + title)
curl -sL -A "Mozilla/5.0 (compatible; aeo-check)" https://yoursite.com/ | wc -c
curl -sL -A "Mozilla/5.0 (compatible; aeo-check)" https://yoursite.com/ | grep -o '<title>[^<]*</title>'

# 2. Your robots.txt, grepped for the AI crawler names and disallows
curl -s https://yoursite.com/robots.txt | grep -Ei "gptbot|oai-searchbot|claudebot|claude-searchbot|perplexitybot|google-extended|user-agent|disallow"

# 3. Does the index file exist? (404 is fine — it is optional)
curl -s -o /dev/null -w "%{http_code}\n" https://yoursite.com/llms.txt

How to read the results: a small byte count with no title on command 1 means your page is a JavaScript shell — that is your top problem. A Disallow: / under User-agent: * in command 2, with no crawler-specific rules, means you are blocking every unnamed agent. A 200 on command 3 is a start, not a finish — the file still has to parse as the spec’s format.

If you’d rather spend the next twenty minutes doing this properly across your top pages, your first 20 minutes of AEO walks through the full manual audit — robots.txt, fetch checks, llms.txt, structured data, and the two-surface question probe — with what each step can and cannot tell you. The whole series, in reading order, is on the AEO/GEO basics hub page.