Crawl Budget

Crawl budget is the daily quota of fetches a search engine will spend on your site. Two dials set it: crawl capacity (what your infrastructure tolerates) and crawl demand (how much the engine wants your content). New sites and low-authority sites get small budgets; popular, frequently-updated sites get large ones.

When it actually matters

The rule: crawl budget problems are URL-space problems. Symptoms:

• Faceted filters generating millions of combinations (?color=red&size=m&sort=price...)
• Calendar pages with infinite next-month links
• Staging or duplicate paths left crawlable
• Soft-404s and expired pages returning 200

Each junk URL crawled is a good URL not crawled — on a big site, that’s indexation loss on the pages that earn money.

When it doesn’t matter

Under ~10k URLs, clean architecture, no accidental URL explosion: Google crawls what exists and the “budget” never binds. Spending a sprint on crawl-budget optimization for a blog with 300 posts is time stolen from content and links.

How to measure honestly

Server logs are the only source of truth. Filter hits by verified Googlebot (reverse-DNS check — spoofed user-agents are common), count by day, then group by URL pattern. The deliverable is a one-table answer: which URL families consume what share of fetches, and whether that share matches their value.

Related concepts

• Topical Authority — clean URL space is a precondition for cluster signals
• SERP Difficulty — unindexed pages have infinite difficulty

FAQ

What is crawl budget in SEO?

Crawl budget is the number of URLs a search engine crawler will request from your site per time period. It is set by the engine from two factors: crawl capacity (how much load your server can take) and crawl demand (how much the engine wants your pages). Wasted on junk URLs, it starves the pages that matter.

Does crawl budget matter for small sites?

Rarely. Under roughly 10,000 unique URLs with sane internal linking, Google crawls what it wants and budget is not a constraint. It becomes a real problem with faceted navigation, parameter URLs, infinite internal-link chains, or sites that bloat their URL space accidentally.

How do you check crawl budget?

From server logs, not tools: filter for Googlebot, count daily hits, verify the reverse DNS (fake Googlebots are common), then break down which URL patterns consume the hits. Our audit tooling classifies log-extracted URLs so you can see spend on valuable vs junk pages.