Crawl budget: what it is and when you need to manage it

Google's crawl budget guide is aimed at large sites, giving examples of sites with more than a million unique pages that change moderately often, or more than ten thousand pages that change very frequently, and sites with many URLs reported as 'Discovered, currently not indexed'. Most small and medium sites are crawled fully without any management. On those sites, indexing problems usually stem from quality, duplication, or linking.

These patterns make Googlebot spend requests on URLs that do not matter.

In Search Console, open Settings, then Crawl stats. It shows total crawl requests, download size, average response time, and breakdowns by response code, file type, purpose (discovery or refresh), and Googlebot type. Rising average response time or a growing share of errors reduces crawl capacity. A large share of requests going to parameter URLs or redirects shows waste. Host status flags robots.txt, DNS, or server problems.

Make the server fast and stable, since Google raises crawl capacity for responsive sites. Remove or block low-value URL spaces. Keep sitemaps limited to canonical, indexable URLs with accurate lastmod. Link important pages prominently so they get crawl demand. Return 404 or 410 for removed content rather than soft 404s. Avoid using noindex as a crawl budget tool: Google still has to crawl a page to see noindex, so blocking crawling of truly useless URL patterns is the more direct fix.

If your site has fewer than tens of thousands of URLs and new pages are crawled within days, crawl budget is unlikely to be the issue; look at content quality and internal links. serionflow.com is small, so we focus on the quality and internal linking of each page rather than on crawl budget. Our guide to 'Discovered, currently not indexed' covers the overlap between crawl capacity and page priority. SerionFlow is the wrong tool for crawl budget work on a large site; that is server, URL, and log analysis it does not do.