Generator · Free, no signup
robots.txt generator for AI crawlers
Build a robots.txt that says exactly what you mean to the fifteen AI crawlers that matter, with a plain explanation of what each one does.
Every AI crawler you block is an AI answer you cannot appear in. Blocking is a legitimate choice for licensing or copyright reasons, and an expensive one if your goal is visibility. Tick the crawlers you want to allow and copy the result.
The tool
Your results appear here as soon as you change a value.
How to use it
- Decide your position on AI crawlers first Visibility or protection. There is no setting that gives you both, and pretending otherwise is how sites end up invisible by accident.
- Allow the ones you want to be cited by GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot are the ones that put you in answers people read.
- Understand Google-Extended It controls whether your content is used for Gemini, not whether you rank in Google Search. Blocking it does not hurt your rankings, and blocking Googlebot very much does.
- Never use robots.txt as security It is a public request, not access control. Anything genuinely private needs authentication.
How the numbers work
Everything here runs in JavaScript in your browser. No input is sent anywhere, nothing is stored, and there is no account, no email gate and no export limit. The maths is deliberately simple so you can check it: where a benchmark is used, the source is named on the page rather than hidden behind a "based on industry data" claim.
Where this calculator uses published research, it uses these: the position-CTR meta-analysis from First Page Sage (updated May 2025), the 34.5% AI Overview click reduction measured by Ahrefs across 300,000 keywords, and Pew Research Center's click-behaviour study. Tool prices come from vendor pricing pages read on 2 August 2026. All of them are averages, and averages are a starting point, not a forecast: your own Search Console data beats every benchmark on this page.
Frequently asked questions
Should I block AI crawlers?
Only if you have a licensing or copyright reason. A blocked crawler cannot cite you, so blocking is a decision to remove yourself from that engine's answers. Publishers with syndication deals block; most businesses seeking visibility should not.
Does blocking Google-Extended hurt my rankings?
No. Google-Extended governs use of your content for Gemini and related products, not Google Search crawling or ranking. Blocking Googlebot itself is what removes you from Search.
Do AI crawlers respect robots.txt?
The major named ones publish that they do, and independent log analyses generally support it. Nothing enforces it technically, and some scrapers ignore it entirely. Treat robots.txt as a request that reputable operators honour.
Related free tools
llms.txt generator
Build a valid llms.txt file for your site in a minute, and read a straight answer about whether it…
GEO content scorer
Paste a draft and get a structural score against the ten signals that correlate with being quoted…
Schema markup generator
Generate valid JSON-LD structured data for FAQs, articles, how-tos, software, organizations, local…
Read more on this
Last verified 2 August 2026. Prices, plans and features on this page were read on each vendor's own site on that date. Vendors change pricing often, so confirm before you buy. Spotted an error? Tell us and we will fix it.