Advertisement

Unable to Change Googlebot Crawl Rate via Google Webmaster Tools

  • 907
    READS
Unable to Change Googlebot Crawl Rate via Google  Webmaster Tools

Google Webmaster Tools allow you to set the Googlebot crawl rate and make it crawl your site faster. To access the feature:

  1. Sign into Google Webmaster Tools;
  2. Click Settings.
  3. In the Crawl rate section, select the option you want.

I mentioned this feature in my quite popular post on how to increase your site crawl rate. What followed the post was quite a number of comments reporting that the feature was either non-accessible (some people just didn’t have the option to change anything) or incomplete (people could set the bot to crawl slower but not faster).

First of all, I’d like to make it clear that this doesn’t mean anything bad. Google is not dumb at all: it will adjust its crawl rate the best possible way automatically. Citing John Mueller,

You usually won’t have to fiddle with those settings, which is why they aren’t available to all sites. We usually work out how much we can (and want to) crawl automatically. I would generally only change these settings if we’re crawling your site too hard (and slowing down your server).

Some possible reasons why you might be unable to change the crawl rate via Google Webmaster Tools:

  • you can’t change the crawl rate for a sub folder (versus a root domain or sub domain);
  • some large site groups (such as blogger) have this feature pre-set for them;
  • Google has found the most suitable rate for your site (based on your server) and doesn’t want any input into the setting from you at the moment.

Note: also read this great discussion on how crawl rate change effects the site.

CategorySEO
ADVERTISEMENT

Subscribe to SEJ

Get our daily newsletter from SEJ's Founder Loren Baker about the latest news in the industry!

Ebook

Ann Smarty

Brand amd Community Manager at Internet Marketing Ninjas

Ann Smarty is the blogger and community manager at Internet Marketing Ninjas. Ann's expertise in blogging and tools serve as ... [Read full bio]

ADVERTISEMENT
Advertisement
Read the Next Article
Read the Next