AVANGARDISM
SEO and GEO glossary

Crawl budgetCrawl budget

Definition

Crawl budget is the number of pages Google can and wants to crawl on a site within a given time. It depends on the server’s capacity and on how much interest Google has in the pages. According to Google, it only matters for very large sites or sites that change very often.

Understanding it

Google’s guide on the topic is aimed at sites with over a million pages changing weekly, or over 10,000 pages changing daily. A brochure site with a few hundred pages need not worry about it.

On a large site, budget is wasted on useless pages: endless search filters, duplicates, URL parameters, redirect chains. Important pages then get visited less often.

The levers: block crawling of worthless areas, remove duplicates, respond fast and keep the sitemap up to date.

A concrete example

Illustrative example: a listings site

A property listings site has hundreds of thousands of URLs, mostly filter combinations. New listings take days to appear in Google.

Crawling of filters is blocked in robots.txt and the sitemap only lists active listings. New listings get crawled faster.

The common mistake

Worrying about it on a small site

Optimising crawl budget on a 50-page site changes nothing: Google crawls it with ease. On a small site, work on content and links, not crawling.

Frequently asked questions

Crawl budget: your questions

  • According to Google, only if it has tens of thousands of pages changing daily, or hundreds of thousands changing weekly.