Craig Campbell SEO Training & Consultancy

Tutorials

Crawl budget: when it matters for SEO and when it doesn't

By Craig Campbell · 10 Nov 2026

3 min read

Crawl budget is how many URLs a search engine can and will crawl on your site in a given time. It matters for some sites and is a distraction for most. Here's how to tell, and what to do.

It's for site owners and SEO teams who've seen the term in audits. The worked example uses "Northgate Property Listings", a made-up Edinburgh property portal. Its figures are illustrative only, not a client or a measurement. Google's crawl budget documentation was checked on 10 November 2026.

Does crawl budget matter for your site?

Google describes its crawl budget guide as advanced, intended mainly for:

  • large sites (roughly 1 million or more unique pages) whose content changes about weekly;
  • medium or larger sites (roughly 10,000 or more unique pages) whose content changes very rapidly, such as daily;
  • sites where a large share of URLs are listed in Search Console as "Discovered – currently not indexed".

Google calls these rough estimates. If new pages are usually crawled the day they're published, an up-to-date sitemap and the Page indexing report are enough.

Most small business sites, blogs and shops with a few thousand products aren't limited by crawl budget. Indexing problems there usually come from quality, duplication or technical errors.

How Google decides how much to crawl

Google describes two parts:

  • Crawl capacity limit: how much Google can crawl without overloading your server. Fast, reliable sites get more; slow or erroring sites get less.
  • Crawl demand: how much Google wants to crawl, based on popularity, how often pages change and how many URLs it knows.

What wastes crawling

On large sites, crawling is often spent on URLs that should not exist or do not matter:

  • faceted navigation creating huge numbers of filter combinations;
  • session IDs and tracking parameters creating duplicate URLs;
  • endless calendar or search result pages;
  • long redirect chains and soft 404 pages;
  • duplicate content reachable at several addresses.

Worked example (fictional): Northgate has 80,000 listings, but a crawl finds 2.1 million URLs, mostly from combining filters for price, bedrooms and area. New listings are slow to appear in search. The problem is not the size of the listing inventory; it is the URLs the filters create.

The shop filters SEO guide explains how to choose indexable filter pages and distinguish crawling controls from noindex and canonical signals.

What to do if it does matter

  • Consolidate duplicates so each piece of content has one main URL.
  • Block crawling of unimportant URL patterns with robots.txt, such as endless filters. robots.txt controls crawling, not indexing; use noindex on a crawlable page to keep it out of results.
  • Return proper status codes: 404 or 410 for removed pages, not soft 404s.
  • Keep sitemaps current and include only URLs you want indexed, with accurate last-modified dates.
  • Improve server speed and reliability, which affects the crawl capacity limit.
  • Shorten redirect chains so each redirect goes straight to the final page.

Worked example (fictional): Northgate keeps indexable pages only for filter combinations people search for, such as "2 bedroom flats in Leith", and blocks crawling of other combinations. Its sitemap lists only live listings. These are illustrative decisions, not tested outcomes.

How to check

  • Crawl stats (Search Console, under Settings): requests, response times and codes.
  • Page indexing report: which URLs are left out.
  • Server logs, if you have them: where crawling actually goes. The log file analysis guide explains coverage, crawler verification and interpreting the patterns.

Limitations

This doesn't cover other search engines or log-analysis tools in detail. Google can change how it crawls, and these steps improve efficiency rather than guarantee faster indexing. The fictional portal is used only to illustrate decisions.

Your next step and sources

First check if your site fits Google's description. If not, focus on content quality and technical basics. Our technical SEO guide and Search Console guide are good next steps.

Sources checked 10 November 2026:

Let's work together.

Whether you need an SEO partner, a consultant, a speaker for your event or training for your team — get in touch and let's chat.

Get in touch
Get in touch