What Is Crawl Budget?

Google's definition

Google describes crawl budget as two parts combined:

• Crawl capacity limit: how much Google can crawl without overloading your server. It rises when your site responds quickly and falls when it slows down or returns errors.
• Crawl demand: how much Google wants to crawl, driven by how many URLs it thinks you have, how popular they are and how often they change.

Do you need to worry about crawl budget?

Google's crawl budget guide is written for sites with roughly 1 million or more unique pages that change about weekly, sites with 10,000 or more pages that change daily, and sites with a large share of URLs stuck as "Discovered - currently not indexed". Google calls these rough estimates, not exact thresholds.

A new SaaS with a few hundred pages almost never has a crawl budget problem. If its pages are not indexed, the cause is usually discovery, duplication or perceived quality. See indexing.

What wastes crawl budget

• Faceted navigation and filters that create endless URL combinations.
• Duplicate URLs from tracking parameters, session IDs or multiple paths to the same content.
• Soft 404s and long redirect chains.
• Infinite spaces such as calendars with a "next" link forever.
• Large batches of thin pages, a common side effect of careless programmatic SEO.

How to make crawling efficient

Keep server responses fast, consolidate duplicates with canonical tags, block genuinely useless URL patterns in robots.txt, return proper 404 or 410 codes for removed pages, keep your sitemap limited to canonical URLs you want indexed, and link important pages from your navigation. Search Console's Crawl stats report shows how Google is crawling your site.

Illustrative examples

Hypothetical scenarios that show how the term applies. They are not reports of real cases.

• An online store with filter combinations for size, color and price generates millions of URLs; blocking the filter parameters lets Google spend its crawls on product pages.
• A 300-page SaaS site has slow indexing because new posts are not linked from anywhere; adding them to a blog index fixes it, with no crawl budget work needed.