<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Robots-Txt on AECITED — AI visibility: analyzed and fixed monthly</title><link>https://aecited.com/tags/robots-txt/</link><description>Recent content in Robots-Txt on AECITED — AI visibility: analyzed and fixed monthly</description><generator>Hugo</generator><language>en</language><lastBuildDate>Thu, 29 Oct 2026 00:00:00 +0000</lastBuildDate><atom:link href="https://aecited.com/tags/robots-txt/index.xml" rel="self" type="application/rss+xml"/><item><title>Do AI Crawlers Respect robots.txt? What the Experiment Found</title><link>https://aecited.com/posts/ai-crawlers-robots-txt/</link><pubDate>Thu, 29 Oct 2026 00:00:00 +0000</pubDate><guid>https://aecited.com/posts/ai-crawlers-robots-txt/</guid><description>&lt;p>&lt;strong>Officially, the major AI engines say their crawlers respect robots.txt, and each publishes per-bot tokens that let site owners opt out of training and/or search. A controlled experiment reported on r/SEO in September 2026 found a different picture in the logs: at least some AI crawler traffic ignored robots.txt entirely and arrived as Chrome and Safari browser requests. The honest synthesis of the two sides: robots.txt is a permission signal and a courtesy protocol, not a security control — it stops the bots that tell you their name, and if your content must not be quoted, the real control belongs at the content and technical layers, not in the file alone.&lt;/strong>&lt;/p></description></item><item><title>How AI engines crawl your site and what breaks silently</title><link>https://aecited.com/posts/how-ai-engines-crawl-your-site/</link><pubDate>Tue, 20 Oct 2026 00:00:00 +0000</pubDate><guid>https://aecited.com/posts/how-ai-engines-crawl-your-site/</guid><description>&lt;p>&lt;strong>AI answer engines crawl your site in two distinct roles: training crawlers, which fetch content that may feed model training, and &lt;a href="https://aecited.com/posts/aeo-glossary/">grounding&lt;/a> crawlers, which build the indexes that let your pages appear in live search answers.&lt;/strong> The two roles have different crawler identities, different robots.txt tokens, and different costs when you block them — and every one of them fetches over plain HTTP, which means what a crawler sees is what an ordinary HTTP client gets. The most common failure is silent: when a grounding crawler stops reaching your pages, no dashboard, no console and no report tells you.&lt;/p></description></item><item><title>LLM crawlers in 2026: a robots.txt reference</title><link>https://aecited.com/posts/llm-crawlers-reference/</link><pubDate>Mon, 19 Oct 2026 00:00:00 +0000</pubDate><guid>https://aecited.com/posts/llm-crawlers-reference/</guid><description>One row per LLM crawler — Anthropic, OpenAI, Perplexity, Google — with purpose, allow/block rule, and the official documentation behind each. Updated 2026-10-19.</description></item></channel></rss>