Glossary · Technical
Crawl budget
Definition. Crawl budget is the number of URLs a search engine will fetch from your site in a given period, set by how much it wants to crawl you and how much your server can take. It only becomes a practical constraint on large sites, roughly tens of thousands of URLs and up.
On big sites the waste is usually structural: faceted navigation, session parameters, infinite calendars and duplicate variants generating millions of low-value URLs.
Small sites almost never have a crawl budget problem. If a ten-page site is not indexed, the cause is quality, canonicals or blocking, not budget.
Where this shows up on this site
Related terms
Indexation
Indexation is whether a search engine has stored your page and made it eligible to appear in results. A…
Orphan page
An orphan page is a page with no internal links pointing to it. Search engines can only find it through…
robots.txt
robots.txt is a plain-text file at the root of a domain telling crawlers which paths they may request…
Canonical URL
A canonical URL is the version of a page you declare as the original when several URLs serve similar…
More in Technical: Internal linking · Anchor text · Orphan page · Indexation · Canonical URL · Duplicate content · Keyword cannibalization · Core Web Vitals
Frequently asked questions
What does Crawl budget mean?
Crawl budget is the number of URLs a search engine will fetch from your site in a given period, set by how much it wants to crawl you and how much your server can take. It only becomes a practical constraint on large sites, roughly tens of thousands of URLs and up.
Why does Crawl budget matter for AI SEO?
On big sites the waste is usually structural: faceted navigation, session parameters, infinite calendars and duplicate variants generating millions of low-value URLs.
What is the most common mistake with crawl budget?
Small sites almost never have a crawl budget problem. If a ten-page site is not indexed, the cause is quality, canonicals or blocking, not budget.
Last verified 2 August 2026. Prices, plans and features on this page were read on each vendor's own site on that date. Vendors change pricing often, so confirm before you buy. Spotted an error? Tell us and we will fix it.
Put it into practice
Thirty AI SEO tools compared on verified pricing, capabilities and AI search support.