Most AI crawling advice online is a screenshot of somebody’s server log from two years ago, and several of the bots in it don’t exist anymore. This page is the opposite: one table, one row per crawler, and every row tied to the official documentation page that defines that bot — Anthropic’s help center, OpenAI’s crawler docs, Perplexity’s docs, and Google’s crawling infrastructure documentation. All rows were re-verified on 2026-10-19, and when a vendor changes its docs, this page gets changed too. That’s in the update log at the bottom.
If you need the business case for why any of this matters before you touch robots.txt, start with which surface actually cites B2B SaaS sites. If you’re trying to get cited rather than just opting out, read how to get your B2B SaaS product cited by ChatGPT first and come back here for the mechanics.
Training crawlers vs. grounding crawlers: the distinction that decides what you block
There are three kinds of “AI crawler,” and the difference is the entire ballgame:
Training crawlers collect content that may feed model training. OpenAI’s docs say GPTBot “is used to crawl content that may be used in training” its foundation models, and Anthropic’s docs say ClaudeBot collects web content that “could potentially contribute to” model training. Disallowing them is an opt-out signal: your future content stays out of the training data. Nobody publishes what that’s worth in revenue terms — and no one should quote you a number.
Grounding crawlers build an index so your site can appear in live search answers. These are the ones that decide whether you show up in ChatGPT’s search results (OAI-SearchBot), Claude’s search results (Claude-SearchBot), or Perplexity’s (PerplexityBot). Each vendor says, in writing, that this bot does not train models — OpenAI’s OAI-SearchBot is “for search,” Perplexity says PerplexityBot “is not used to crawl content for AI foundation models.”
User-directed fetchers fire when a human asks a question or clicks something. ChatGPT-User, Claude-User, and Perplexity-User all work this way. Here’s the part most people miss: OpenAI’s docs say “robots.txt rules may not apply” to ChatGPT-User, and Perplexity’s docs say Perplexity-User “generally ignores robots.txt rules.” You can’t reliably block a user’s direct request, and you probably don’t want to.
One special case: Google-Extended is not a bot at all. It’s a robots.txt token. Google’s own docs explain that sites can “use Google-Extended in robots.txt to control, among other things, whether their content helps train future versions of Gemini models,” and that “utilizing Google-Extended doesn’t affect a site’s inclusion in Search, nor do we use Google-Extended as a ranking signal.” The actual crawling still happens by Googlebot; the token just controls what Google may do with it for Gemini and Vertex AI training.
The practical takeaway: “block AI” is not one switch. Training and grounding are independent axes per vendor. You can block ClaudeBot and stay fully visible in Claude search. You can allow GPTBot training and still not appear in ChatGPT search. Confusing the axes is how sites silently opt themselves out of the one thing they actually wanted.
The table
Verified 2026-10-19 against the official docs listed in the last column. User-agent names below are the tokens you use in robots.txt.
| Crawler | Who runs it | Purpose | How to allow / block | Source |
|---|---|---|---|---|
| ClaudeBot | Anthropic | Training — collects web content that may contribute to model training. Blocking excludes your future content from training datasets | User-agent: ClaudeBot / Disallow: / in robots.txt (per subdomain) |
Anthropic help center |
| Claude-User | Anthropic | User-directed fetch — retrieves pages when someone asks Claude a question. Blocking may reduce visibility for user-directed search | Same robots.txt pattern | Anthropic help center |
| Claude-SearchBot | Anthropic | Grounding — indexes content to improve Claude search result quality. Blocking may reduce your visibility in Claude search | Same robots.txt pattern | Anthropic help center |
| GPTBot | OpenAI | Training — content may be used to train OpenAI’s foundation models. Disallow = no training use | User-agent: GPTBot / Disallow: / in robots.txt |
OpenAI crawler docs |
| OAI-SearchBot | OpenAI | Grounding — surfaces your site in ChatGPT search answers. Opting out means you won’t appear in ChatGPT search answers, though you can still appear as navigational links. Changes take ~24h | Same robots.txt pattern; IP list at openai.com/searchbot.json | OpenAI crawler docs |
| ChatGPT-User | OpenAI | User-directed fetch for ChatGPT, Custom GPTs, and GPT Actions. Not used for search. “robots.txt rules may not apply” | robots.txt declared, but OpenAI says it may not be honored — don’t rely on it | OpenAI crawler docs |
| OAI-AdsBot | OpenAI | Validates landing pages for ads submitted on ChatGPT. Not used for training | robots.txt; only relevant if you run ChatGPT ads | OpenAI crawler docs |
| PerplexityBot | Perplexity | Grounding — surfaces and links sites in Perplexity search. Explicitly “not used to crawl content for AI foundation models.” Blocked pages may still be indexed as domain + headline + brief factual summary. Changes take up to 24h | Same robots.txt pattern; IP list at perplexity.com/perplexitybot.json | Perplexity crawler docs and help center |
| Perplexity-User | Perplexity | User-directed fetch when a user’s question needs a page. “Generally ignores robots.txt rules” | robots.txt declared, but Perplexity says it generally isn’t honored | Perplexity crawler docs |
| Google-Extended | Not a crawler — a robots.txt product token controlling whether Googlebot-collected content may train future Gemini models and Vertex AI. No effect on Search inclusion or ranking | User-agent: Google-Extended / Disallow: / |
Google crawling infrastructure docs and GSC generative AI control |
This list covers the four vendors whose official documentation we could verify line-by-line on 2026-10-19. You will see other agents in your server logs (xAI, Meta, Cohere, and various smaller crawlers). We deliberately don’t list them here until we’ve verified each row against an official source; the update log records when new rows are added.
What blocking each one actually costs
The table tells you what each bot does. Here’s what the choice is worth to a B2B SaaS site — with the honesty that most of these costs are structural, not quantified:
Blocking OAI-SearchBot is the decision with the most at stake, because ChatGPT is the largest AI referral surface for B2B — our current numbers show 50–63% of AI referrals depending on the source and window. OpenAI’s docs are explicit about the cost: opted-out sites “will not be shown in ChatGPT search answers,” though brand searches can still produce a navigational link. You keep brand traffic, you lose discovery.
Blocking Claude-SearchBot costs you a smaller surface. Claude’s share of AI referrals has been single digits in every monthly window we’ve tracked (roughly 0.9–2.7%, window-dependent — the full history is in the traffic numbers post). Whether that’s material depends on where your buyers ask questions.
Blocking PerplexityBot costs you Perplexity appearances, with one wrinkle worth knowing: per Perplexity’s help center (last modified 2026-09-14), a blocked page “may still [have] the domain, headline, and a brief factual summary” indexed. A block isn’t necessarily total silence.
Blocking GPTBot or ClaudeBot (training) has no documented visibility cost — no vendor says blocking training removes you from search answers. What you’re giving up is a claim about the future: you don’t know how much pre-trained knowledge of your category will matter in 18 months. No vendor publishes that number, and neither do we. Treat it as a policy decision (is your content IP you don’t want in training data?), not a traffic decision.
Blocking Google-Extended is the cheapest trade on this list: Google states in writing that it “doesn’t affect a site’s inclusion in Search” and isn’t a ranking signal. You opt out of Gemini training and lose nothing in the search results that pay your bills.
The per-surface breakdown — which surface is worth fighting for, with the evidence for each — is in which surface actually cites B2B SaaS sites.
How to check your robots.txt isn’t silently blocking the wrong thing
The most common real-world failure isn’t a missing rule. It’s one of these:
- A blanket
User-agent: */Disallow: /. If you (or a staging environment, a password-protected area, or a plugin) disallow everything for*, and you have no specific block for ClaudeBot, GPTBot, or OAI-SearchBot, the wildcard rule applies to them too. You’re then blocking the grounding crawlers without ever writing a rule that names them. This is the single most common “we never opted out, but nothing shows up in ChatGPT” case. - Subdomains. Anthropic’s docs are explicit: add the rule “for every subdomain that you wish to opt out from.” If your docs live on docs.yourproduct.com and your robots.txt is on yourproduct.com, the two files are independent.
- The WAF is the real gate. Perplexity’s docs include WAF configuration steps (Cloudflare, AWS) because firewalls that block unknown user agents or non-listed IPs will drop a bot that robots.txt would have allowed — and vice versa. Check both layers.
- Latency. OpenAI says robots.txt changes take “about 24 hours” to take effect in ChatGPT search; Perplexity says “up to 24 hours.” If you changed the file today, judge results next week, not next hour.
- Verify by IP, not by hope. OpenAI publishes IP ranges at openai.com/searchbot.json, openai.com/gptbot.json, and openai.com/chatgpt-user.json; Perplexity publishes perplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json; Anthropic’s help center lists its crawler source IPs. If you allow a bot, confirm the requests are actually arriving (they’re in your server logs, with the exact user-agent strings in the table). If you block one, confirm the requests stop.
- Don’t confuse robots.txt with the other levers. For Google, the GSC “Search generative AI” control (Settings in Search Console) controls whether your content is shown in AI Overviews and AI Mode — that’s separate from Google-Extended, which is the training token. The mechanics of both are in how to measure AI search traffic in 2026, and our take on what llms.txt adds (or doesn’t) is in llms.txt v2.
Quick check before you commit
The fastest way to see which of these crawlers are already on your site — and which surfaces you already appear on — is the free quick check. Fifteen minutes, no login, and it tells you whether the robots.txt conversation above is even your problem yet.
If you want the behind-the-scenes on how we source and date every number on this page (including the crawler rows), it’s all in the methodology.
Update log
2026-10-19 — initial edition. Ten rows, all verified against official documentation the same day: Anthropic’s help center article on its three crawlers (article dated 2026-04-08), OpenAI’s “Overview of OpenAI Crawlers” (developers.openai.com, retrieved 2026-10-19; four user agents listed, including OAI-AdsBot), Perplexity’s crawler docs (docs.perplexity.ai, retrieved 2026-10-19) and its help center article on robots.txt compliance (last modified 2026-09-14), and Google’s crawling infrastructure docs plus the GSC “Search generative AI” control help article (retrieved 2026-10-19). No row is marked unverified. When a vendor’s docs change — a new bot, a changed robots.txt behavior, a renamed user agent — this page will be updated and this log will say what changed and when.