Crawl budget is how many URLs a search engine can and will crawl on your site in a given time. It matters for some sites and is a distraction for most. Here's how to tell, and what to do.
It's for site owners and SEO teams who've seen the term in audits. The worked example uses "Northgate Property Listings", a made-up Edinburgh property portal. Its figures are illustrative only, not a client or a measurement. Google's crawl budget documentation was checked on 10 November 2026.
Does crawl budget matter for your site?
Google describes its crawl budget guide as advanced, intended mainly for:
- large sites (roughly 1 million or more unique pages) whose content changes about weekly;
- medium or larger sites (roughly 10,000 or more unique pages) whose content changes very rapidly, such as daily;
- sites where a large share of URLs are listed in Search Console as "Discovered – currently not indexed".
Google calls these rough estimates. If new pages are usually crawled the day they're published, an up-to-date sitemap and the Page indexing report are enough.
Most small business sites, blogs and shops with a few thousand products aren't limited by crawl budget. Indexing problems there usually come from quality, duplication or technical errors.
How Google decides how much to crawl
Google describes two parts:
- Crawl capacity limit: how much Google can crawl without overloading your server. Fast, reliable sites get more; slow or erroring sites get less.
- Crawl demand: how much Google wants to crawl, based on popularity, how often pages change and how many URLs it knows.
What wastes crawling
On large sites, crawling is often spent on URLs that should not exist or do not matter:
- faceted navigation creating huge numbers of filter combinations;
- session IDs and tracking parameters creating duplicate URLs;
- endless calendar or search result pages;
- long redirect chains and soft 404 pages;
- duplicate content reachable at several addresses.
Worked example (fictional): Northgate has 80,000 listings, but a crawl finds 2.1 million URLs, mostly from combining filters for price, bedrooms and area. New listings are slow to appear in search. The problem is not the size of the listing inventory; it is the URLs the filters create.
The shop filters SEO guide explains how to choose indexable filter pages and distinguish crawling controls from noindex and canonical signals.
What to do if it does matter
- Consolidate duplicates so each piece of content has one main URL.
- Block crawling of unimportant URL patterns with robots.txt, such as endless filters. robots.txt controls crawling, not indexing; use noindex on a crawlable page to keep it out of results.
- Return proper status codes: 404 or 410 for removed pages, not soft 404s.
- Keep sitemaps current and include only URLs you want indexed, with accurate last-modified dates.
- Improve server speed and reliability, which affects the crawl capacity limit.
- Shorten redirect chains so each redirect goes straight to the final page.
Worked example (fictional): Northgate keeps indexable pages only for filter combinations people search for, such as "2 bedroom flats in Leith", and blocks crawling of other combinations. Its sitemap lists only live listings. These are illustrative decisions, not tested outcomes.
How to check
- Crawl stats (Search Console, under Settings): requests, response times and codes.
- Page indexing report: which URLs are left out.
- Server logs, if you have them: where crawling actually goes. The log file analysis guide explains coverage, crawler verification and interpreting the patterns.
Limitations
This doesn't cover other search engines or log-analysis tools in detail. Google can change how it crawls, and these steps improve efficiency rather than guarantee faster indexing. The fictional portal is used only to illustrate decisions.
Your next step and sources
First check if your site fits Google's description. If not, focus on content quality and technical basics. Our technical SEO guide and Search Console guide are good next steps.
Sources checked 10 November 2026:

