Glossary · Technical
robots.txt
Definition. robots.txt is a plain-text file at the root of a domain telling crawlers which paths they may request. It controls crawling, not indexing, and it is a request that well-behaved bots honour rather than an access control that enforces anything.
A URL blocked in robots.txt can still be indexed if other pages link to it, because Google can index a URL it has never fetched. To keep a page out of the index, use a noindex directive on a crawlable page.
It is now also the main lever for AI crawlers. Each one you disallow is an engine whose answers you cannot appear in.
Never use it to hide anything sensitive: the file is public and lists exactly what you did not want found.
Where this shows up on this site
Related terms
AI crawler
An AI crawler is a bot that fetches web pages on behalf of an AI company, either to build training data…
noindex
noindex is a directive, in a meta robots tag or an X-Robots-Tag HTTP header, telling search engines not…
Crawl budget
Crawl budget is the number of URLs a search engine will fetch from your site in a given period, set by…
llms.txt
llms.txt is a proposed markdown file at the root of a domain listing the pages an AI system should read…
More in Technical: Internal linking · Anchor text · Orphan page · Crawl budget · Indexation · Canonical URL · Duplicate content · Keyword cannibalization
Frequently asked questions
What does robots.txt mean?
robots.txt is a plain-text file at the root of a domain telling crawlers which paths they may request. It controls crawling, not indexing, and it is a request that well-behaved bots honour rather than an access control that enforces anything.
Why does robots.txt matter for AI SEO?
A URL blocked in robots.txt can still be indexed if other pages link to it, because Google can index a URL it has never fetched. To keep a page out of the index, use a noindex directive on a crawlable page.
What is the most common mistake with robots.txt?
Never use it to hide anything sensitive: the file is public and lists exactly what you did not want found.
Last verified 2 August 2026. Prices, plans and features on this page were read on each vendor's own site on that date. Vendors change pricing often, so confirm before you buy. Spotted an error? Tell us and we will fix it.
Put it into practice
Thirty AI SEO tools compared on verified pricing, capabilities and AI search support.