<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Crawlers on AECITED — AI visibility: analyzed and fixed monthly</title><link>https://aecited.com/tags/crawlers/</link><description>Recent content in Crawlers on AECITED — AI visibility: analyzed and fixed monthly</description><generator>Hugo</generator><language>en</language><lastBuildDate>Thu, 29 Oct 2026 00:00:00 +0000</lastBuildDate><atom:link href="https://aecited.com/tags/crawlers/index.xml" rel="self" type="application/rss+xml"/><item><title>Do AI Crawlers Respect robots.txt? What the Experiment Found</title><link>https://aecited.com/posts/ai-crawlers-robots-txt/</link><pubDate>Thu, 29 Oct 2026 00:00:00 +0000</pubDate><guid>https://aecited.com/posts/ai-crawlers-robots-txt/</guid><description>&lt;p>&lt;strong>Officially, the major AI engines say their crawlers respect robots.txt, and each publishes per-bot tokens that let site owners opt out of training and/or search. A controlled experiment reported on r/SEO in September 2026 found a different picture in the logs: at least some AI crawler traffic ignored robots.txt entirely and arrived as Chrome and Safari browser requests. The honest synthesis of the two sides: robots.txt is a permission signal and a courtesy protocol, not a security control — it stops the bots that tell you their name, and if your content must not be quoted, the real control belongs at the content and technical layers, not in the file alone.&lt;/strong>&lt;/p></description></item><item><title>Technical SEO Hygiene Checklist: Which 'Limits' Are Google Rules?</title><link>https://aecited.com/posts/seo-technical-hygiene-checklist/</link><pubDate>Thu, 29 Oct 2026 00:00:00 +0000</pubDate><guid>https://aecited.com/posts/seo-technical-hygiene-checklist/</guid><description>&lt;p>&lt;strong>The technical SEO hygiene baseline in 2026 is a two-layer list: first fix discovery and indexing (robots.txt, sitemap, internal links, redirect hygiene), then fix presentation (title, meta description, image alt text, heading structure, URL structure, anchor text) — and know that most of the famous numbers in that list (60-character titles, 160-character descriptions, images under 200 KB, 3–8-word anchors, redirect chains under 2) are conventions, not Google rules.&lt;/strong> The asks Google actually documents are qualitative: descriptive, relevant, human-readable, &amp;ldquo;shorter is better.&amp;rdquo; Follow the numbers as display heuristics, because they are calibrated to how results render — but do not believe that a broken checklist item is what is holding you back, unless the item is one of the few Google actually owns. That distinction is what this post exists for.&lt;/p></description></item><item><title>How AI engines crawl your site and what breaks silently</title><link>https://aecited.com/posts/how-ai-engines-crawl-your-site/</link><pubDate>Tue, 20 Oct 2026 00:00:00 +0000</pubDate><guid>https://aecited.com/posts/how-ai-engines-crawl-your-site/</guid><description>&lt;p>&lt;strong>AI answer engines crawl your site in two distinct roles: training crawlers, which fetch content that may feed model training, and &lt;a href="https://aecited.com/posts/aeo-glossary/">grounding&lt;/a> crawlers, which build the indexes that let your pages appear in live search answers.&lt;/strong> The two roles have different crawler identities, different robots.txt tokens, and different costs when you block them — and every one of them fetches over plain HTTP, which means what a crawler sees is what an ordinary HTTP client gets. The most common failure is silent: when a grounding crawler stops reaching your pages, no dashboard, no console and no report tells you.&lt;/p></description></item><item><title>LLM crawlers in 2026: a robots.txt reference</title><link>https://aecited.com/posts/llm-crawlers-reference/</link><pubDate>Mon, 19 Oct 2026 00:00:00 +0000</pubDate><guid>https://aecited.com/posts/llm-crawlers-reference/</guid><description>One row per LLM crawler — Anthropic, OpenAI, Perplexity, Google — with purpose, allow/block rule, and the official documentation behind each. Updated 2026-10-19.</description></item></channel></rss>