Topic

Crawlers posts

Technical SEO Hygiene Checklist: Which 'Limits' Are Google Rules?

The technical SEO hygiene baseline: a nine-item checklist separating Google's official asks from the popular 'limits' that are really conventions.

Do AI Crawlers Respect robots.txt? What the Experiment Found

Did AI crawlers respect robots.txt in a September 2026 experiment? What the official docs promise, and how to control what AI can quote.

How AI engines crawl your site and what breaks silently

How AI answer engines fetch your site: training crawlers vs answer crawlers, the plain HTTP fetch path, and what your robots.txt silently does.

LLM crawlers in 2026: a robots.txt reference

Every LLM crawler in one table — who runs it, what it trains, how to allow or block it, and the official doc that proves it. Verified 2026-10-19.

AI Visibility Audit · $99One PDF in 3 business days
Book it