In short
Crawl budget is the number of URLs a search engine will fetch from your site in a period, determined by how much crawling your server can handle and how much the content justifies. It is a genuine constraint only on large sites — typically tens of thousands of URLs or more, or sites generating URLs dynamically. Below that, pages not being indexed is almost always a quality issue rather than a crawling one.
What it actually is
Crawl budget is the number of URLs a search engine will fetch from your site in a given period. It emerges from two things.
Crawl capacity — how much your server can handle without degrading. If responses slow down or errors appear, crawling backs off.
Crawl demand — how much the content justifies fetching. Popular, frequently updated, high-quality pages get crawled more. Pages nothing links to and nothing changes get crawled rarely.
Does it apply to you?
For most sites, no.
Google's own guidance is that sites below roughly a few thousand URLs are generally crawled adequately, and crawl budget is not the reason their pages are missing.
It becomes a genuine constraint when you have tens of thousands of URLs, when URLs are generated dynamically at scale — faceted navigation, calendars, search results — or when the server is slow enough that crawling is throttled.
If you have four hundred pages and some are not indexed, the cause is quality rather than crawling. Optimising crawl budget will change nothing, and it is a popular way to avoid the harder answer.
What wastes it
On sites where it does apply, the waste is usually concentrated in a few patterns.
Faceted navigation. Filter combinations producing thousands of near-identical URLs. Reliably the largest source on catalogue sites.
Parameters that do not change content. Sorting, view modes, tracking codes, session identifiers. Each variation is a separate URL to fetch.
Infinite spaces. Calendars generating pages indefinitely, or paginated archives with no end.
Internal redirect chains. Every hop is a fetch. A site whose internal links point at redirects doubles the crawling needed to reach the same content.
Soft 404s. Pages returning 200 with no real content are crawled and re-crawled as though they were valid.
Duplicate content at scale. Templated pages that are near-identical get fetched and discarded repeatedly.
Slow responses. The most direct lever, since crawl capacity is a function of server performance.
What to do about it
Block what should never be crawled. robots.txt prevents fetching, which is what saves budget — note that noindex does not, since the page must be fetched to read the directive. Use robots.txt for infinite spaces and parameter noise; use noindex for pages that should exist for users but not appear in results.
Fix internal redirect chains. Point internal links at final URLs. This is usually quick and has an immediate effect.
Return proper status codes. Real 404s for missing pages, 410 for permanently removed ones.
Keep the sitemap clean. Only canonical, indexable URLs, with accurate last-modified dates so crawlers can prioritise what changed.
Consolidate near-duplicates. Fewer, stronger pages are crawled more efficiently than many weak ones — and this fixes the underlying quality problem rather than only the crawling symptom.
Improve server response time, which raises capacity directly.
Reading the crawl stats
Search Console's crawl stats report shows requests over time, average response time, and a breakdown by response code and file type.
What to look for: rising response times, which reduce capacity; a high proportion of non-200 responses, which is wasted fetching; and crawling concentrated on unimportant URL patterns, which tells you where the waste is.
A useful sanity check is comparing pages crawled against pages you actually want indexed. A large gap points at generated URLs you did not intend to expose.
The common misdiagnosis
Teams frequently conclude they have a crawl budget problem when Search Console shows many URLs as Discovered — currently not indexed.
That status can indicate crawl prioritisation. But on a site of moderate size it more often means the opposite of what people assume: Google has seen the URLs and does not consider them worth fetching, usually because the site recently produced a large number of low-value pages.
The fix there is not to increase crawling. It is to have fewer, better pages — which reduces the URL count and raises crawl demand at the same time.
The short version
If you have fewer than a few thousand URLs, this is not your problem and effort spent here is displaced from the thing that is.
If you are large or generate URLs dynamically, the highest-value actions are blocking infinite and parameter spaces in robots.txt, fixing internal redirect chains, keeping the sitemap accurate, and consolidating near-duplicates.
If you are unsure which situation you are in, count your indexable URLs and compare it with what you intended to publish — book a call if that number surprises you.
Common questions
What is crawl budget?
The number of URLs a search engine will fetch from your site in a period, determined by crawl capacity — how much your server handles without degrading — and crawl demand, meaning how much the content justifies fetching. Popular, frequently updated pages get crawled more.
Does crawl budget matter for my site?
Only if you have tens of thousands of URLs, generate URLs dynamically through facets or calendars, or have a server slow enough to throttle crawling. Below a few thousand URLs it is not the reason pages are missing — that is almost always a quality issue instead.
What wastes crawl budget?
Faceted navigation producing thousands of near-identical URLs, parameters that do not change content, infinite spaces like open-ended calendars, internal redirect chains where every hop is a fetch, soft 404s returning 200, duplicate templated content, and slow server responses.
Does noindex save crawl budget?
No — the page still has to be fetched for the directive to be read. Use robots.txt to prevent crawling of infinite spaces and parameter noise, and reserve noindex for pages that should exist for users but not appear in search results.
Why are my pages 'Discovered — currently not indexed'?
It can indicate crawl prioritisation, but on a moderately sized site it more often means Google has seen the URLs and does not consider them worth fetching — typically after a large number of low-value pages were published. The fix is fewer, better pages rather than more crawling.
