Google Webmaster Tools allow you to set the Googlebot crawl rate and make it crawl your site faster. To access the feature:
- Sign into Google Webmaster Tools;
- Click Settings.
- In the Crawl rate section, select the option you want.
I mentioned this feature in my quite popular post on how to increase your site crawl rate. What followed the post was quite a number of comments reporting that the feature was either non-accessible (some people just didn’t have the option to change anything) or incomplete (people could set the bot to crawl slower but not faster).
First of all, I’d like to make it clear that this doesn’t mean anything bad. Google is not dumb at all: it will adjust its crawl rate the best possible way automatically. Citing John Mueller,
You usually won’t have to fiddle with those settings, which is why they aren’t available to all sites. We usually work out how much we can (and want to) crawl automatically. I would generally only change these settings if we’re crawling your site too hard (and slowing down your server).
Some possible reasons why you might be unable to change the crawl rate via Google Webmaster Tools:
- you can’t change the crawl rate for a sub folder (versus a root domain or sub domain);
- some large site groups (such as blogger) have this feature pre-set for them;
- Google has found the most suitable rate for your site (based on your server) and doesn’t want any input into the setting from you at the moment.
Note: also read this great discussion on how crawl rate change effects the site.
