Try Now!
Advertisement

Google: Disallowed URLs Do Not Affect Crawl Budget

  • 301
    SHARES
  • 2.5K
    READS
Google: Disallowed URLs Do Not Affect Crawl Budget

Google’s Gary Illyes updated his original writeup on crawl budget with clarification about disallowed URLs.

The document now includes the following information:

“Q: Do URLs I disallowed through robots.txt affect my crawl budget in any way?

A: No, disallowed URLs do not affect the crawl budget.”

The question refers to the “User-agent: * Disallow: /” protocol in robots.txt that blocks web crawlers.

It can either be used to block an entire site from being crawled, or it can be used to block specific URLs from being crawled.

According to the update from Illyes, blocking specific URLs will not impact the crawl budget throughout the rest of the site.

Pages will not get crawled more frequently as a result of other pages on the site being disallowed from crawling.

There’s also no disadvantage to disallowing URLs when it comes to crawl budget.

The updated information appears at the bottom of this article, which is a Webmaster Central blog post from 2017.

Illyes said on Twitter that there are plans to turn the blog post into an official help center article.

More Resources

CategoryNewsSEO
ADVERTISEMENT

Subscribe to SEJ

Get our daily newsletter from SEJ's Founder Loren Baker about the latest news in the industry!

Ebook

Matt Southern

Lead News Writer at Search Engine Journal

Matt Southern has been the lead news writer at Search Engine Journal since 2013. With a degree in communications, Matt ... [Read full bio]

ADVERTISEMENT
Advertisement
Read the Next Article
Read the Next