Crawl budget

Google describes crawl budget as the combination of two things: the crawl capacity limit, how many requests it can make without overloading your server, and crawl demand, how much it wants to crawl based on how popular and fresh your pages are. Together they determine how many pages get fetched.

Google's own guidance is that crawl budget is mainly a concern for very large sites or sites whose content changes very frequently. A typical small business site with a few hundred pages is crawled comfortably, and its indexing problems usually lie elsewhere.

How it works

If a server responds quickly and without errors, Googlebot can crawl more in parallel; if it slows down or returns server errors, Googlebot backs off. Separately, pages that are popular, linked from many places or updated often are crawled more, while stale or low-value URLs are crawled less.

Budget gets wasted on URLs that should not exist: endless filter combinations, session IDs in addresses, calendar pages, duplicates and long redirect chains. Cutting those through robots.txt rules, consistent canonicals and clean internal links lets crawlers spend their visits on pages that matter. Faster servers and fewer errors make more crawling possible, but only demand makes it happen. Speed alone will not bring extra visits to pages Google does not consider worth refreshing, which is why reducing wasted URLs usually helps more than upgrading hosting. Google sets these limits automatically, and site owners cannot ask it to crawl more overall.

Crawl demand responds to signals you control. Pages that are linked from many places, listed in an accurate sitemap and updated with real changes are crawled more often. Large numbers of near-identical pages lower Google's estimate of how useful crawling the site is. Moving a site to a new domain or making large structural changes temporarily increases demand while Google reprocesses everything, which is one reason migrations need a fast, stable server.

Example

An online shop with tens of thousands of products lets visitors filter by size, colour, brand and price, and each combination creates a new URL. Search Console's crawl stats show Googlebot spending most of its time on filtered pages while new products wait weeks to be discovered. Blocking crawling of the filter parameters and linking new products from category pages fixes the backlog.

The shop checks the effect in Crawl stats, which breaks requests down by response code, file type and purpose: discovery of new URLs or refresh of known ones. After the change, a larger share of requests goes to product and category pages and fewer to parameter URLs. URL Inspection on new products shows them being crawled within days, and the Page indexing report shows fewer discovered but not indexed pages.

Why it matters for your business

For large shops, marketplaces and publishers, wasted crawl budget means new or changed pages take longer to appear in search, which costs sales. For most small sites it is not worth worrying about, and spending money on "crawl budget optimisation" there is usually a sign of being sold the wrong service. If a site is small and pages are indexed within days of publishing, crawl budget is not your problem.

Common mistakes

• Worrying about crawl budget on a small site whose real problem is content or links.
• Allowing filters, sorting and tracking parameters to generate unlimited crawlable URLs.
• Using noindex to save crawl budget, when noindexed pages still have to be crawled to be seen.
• Ignoring slow server responses and errors, which make Google crawl less.

Questions people ask

Do small websites need to worry about crawl budget?

Usually not. Google has said crawl budget is mainly an issue for very large sites or those with rapidly changing content. If a small site has pages that are not indexed, the cause is more likely quality, duplication, blocking rules or missing internal links than a shortage of crawling.

How do I check my crawl budget?

Google does not show a single budget figure, but the Crawl stats report in Search Console shows how many requests Googlebot makes per day, response times, status codes and which file types it fetches. A large share of requests to filters, redirects or errors suggests crawling is being wasted.

How can I improve crawl budget?

Remove or block low-value URLs such as endless filter combinations, fix redirect chains and broken links, keep the server fast and reliable, use consistent canonicals and keep your XML sitemap limited to indexable pages. Linking important pages well tells Google where to focus.