Glossary · AI search
AI crawler
Definition. An AI crawler is a bot that fetches web pages on behalf of an AI company, either to build training data or to retrieve current information for an answer. The main named ones include GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended and Applebot-Extended.
The distinction that matters is training versus retrieval. Blocking a training crawler keeps your content out of a future model; blocking a retrieval crawler keeps you out of answers being generated today.
Google-Extended is frequently misunderstood: it governs use of your content in Gemini, not Googlebot's crawling for Search. Blocking it does not affect your rankings.
robots.txt is a request, not enforcement. Reputable operators publish that they honour it; nothing compels them to.
Where this shows up on this site
Related terms
robots.txt
robots.txt is a plain-text file at the root of a domain telling crawlers which paths they may request…
llms.txt
llms.txt is a proposed markdown file at the root of a domain listing the pages an AI system should read…
Crawl budget
Crawl budget is the number of URLs a search engine will fetch from your site in a given period, set by…
More in AI search: AI SEO · Generative engine optimization (GEO) · Answer engine optimization (AEO) · AI Overview · Zero-click search · LLM visibility · Prompt tracking · Citation (AI)
Frequently asked questions
What does AI crawler mean?
An AI crawler is a bot that fetches web pages on behalf of an AI company, either to build training data or to retrieve current information for an answer. The main named ones include GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended and Applebot-Extended.
Why does AI crawler matter for AI SEO?
The distinction that matters is training versus retrieval. Blocking a training crawler keeps your content out of a future model; blocking a retrieval crawler keeps you out of answers being generated today.
What is the most common mistake with ai crawler?
robots.txt is a request, not enforcement. Reputable operators publish that they honour it; nothing compels them to.
Last verified 2 August 2026. Prices, plans and features on this page were read on each vendor's own site on that date. Vendors change pricing often, so confirm before you buy. Spotted an error? Tell us and we will fix it.
Put it into practice
Thirty AI SEO tools compared on verified pricing, capabilities and AI search support.