What Is Crawl Budget? How Google Prioritises Your Site

Crawl budget is the number of URLs Googlebot will crawl on a website within a given period of time. Google allocates a finite crawling capacity to each site based on the site's authority and server capability. That capacity is shared across all URLs Googlebot discovers on the site.

The Two Factors That Set Your Crawl Budget

Crawl rate limit is how fast Googlebot can crawl without overloading the server, controlled by server response time and stability. Crawl demand is how much Google wants to crawl the site, influenced by authority, URL popularity, and content freshness. The combination of these two factors determines the allocated crawl budget.

What Wastes Crawl Budget on E-commerce Sites

The six most common crawl budget wasters are: faceted filter URLs without canonical or noindex control, out-of-stock products returning 200 status with thin content, paginated listing pages beyond page 3-4, duplicate title pages, redirect chains, and soft 404 pages showing empty results with 200 status codes.

How to Prioritise Your Crawl Budget

Canonicalise or block lower-tier URLs so crawl budget concentrates on high-value pages. Priority order: homepage and core service pages, top category pages, product detail pages and blog content, then filter URLs and pagination. Exclude out-of-stock and filter URLs from the XML sitemap.

SEO Implications

Crawl budget constraints can indirectly suppress rankings by delaying or preventing indexation of important pages. The Coverage report in Google Search Console reveals crawl budget problems through large numbers of pages in "Crawled - currently not indexed" or "Discovered - currently not indexed" categories.