Google Webmaster Tools allow you to set the Googlebot crawl rate and make it crawl your site faster. To access the feature:
- Sign into Google Webmaster Tools;
- Click Settings.
- In the Crawl rate section, select the option you want.
I mentioned this feature in my quite popular post on how to increase your site crawl rate. What followed the post was quite a number of comments reporting that the feature was either non-accessible (some people just didn’t have the option to change anything) or incomplete (people could set the bot to crawl slower but not faster).
First of all, I’d like to make it clear that this doesn’t mean anything bad. Google is not dumb at all: it will adjust its crawl rate the best possible way automatically. Citing John Mueller,
You usually won’t have to fiddle with those settings, which is why they aren’t available to all sites. We usually work out how much we can (and want to) crawl automatically. I would generally only change these settings if we’re crawling your site too hard (and slowing down your server).
Some possible reasons why you might be unable to change the crawl rate via Google Webmaster Tools:
- you can’t change the crawl rate for a sub folder (versus a root domain or sub domain);
- some large site groups (such as blogger) have this feature pre-set for them;
- Google has found the most suitable rate for your site (based on your server) and doesn’t want any input into the setting from you at the moment.
Note: also read this great discussion on how crawl rate change effects the site.
what if…. there was more than one bot.. one for webmaster tools and one (several – i.e. cloud), that has been running and indexing sites all along which is how google got the largest index footprint in the first place??
What are the pros and cons for a faster crawl rate? I mean if slows your server down that seems like its not worth increasing the crawl rate!
@Eric, good question :) It is believed that the bot “crawls on a budget”, that is the faster it crawls, the MORE (the deeper) it crawls and the more the index is updated. The latter usually results in better rankings. I have also linked to an article (bottom of the post) where the actual experience with faster crawl rate is described.
@Ann, Thanks for the clarification! I’ll check out the other article and prolly have more questions about that as well!
Thanks!
New content on a regular basis helps. I have a couple of blogs where I post at least once a day, and they get cached within minutes of my posts.
Hmm… my site is crawled very very slow.. i can’t change the cralw rate… what to do… :(
Get rid of the noarchive tag for starters.
This one would suit you better….