Comments on: Unable to Change Googlebot Crawl Rate via Google Webmaster Tools

By: Martin

Martin — Sat, 08 Aug 2009 08:10:29 +0000

Get rid of the noarchive tag for starters.

This one would suit you better….

By: www.eng-rom.com

www.eng-rom.com — Sat, 31 Jan 2009 07:37:01 +0000

Hmm… my site is crawled very very slow.. i can’t change the cralw rate… what to do… :(

By: Rob

Rob — Wed, 28 Jan 2009 01:39:02 +0000

New content on a regular basis helps. I have a couple of blogs where I post at least once a day, and they get cached within minutes of my posts.

By: Eric

Eric — Mon, 26 Jan 2009 17:43:17 +0000

@Ann, Thanks for the clarification! I’ll check out the other article and prolly have more questions about that as well!

Thanks!

By: Ann Smarty

Ann Smarty — Mon, 26 Jan 2009 17:20:47 +0000

@Eric, good question :) It is believed that the bot “crawls on a budget”, that is the faster it crawls, the MORE (the deeper) it crawls and the more the index is updated. The latter usually results in better rankings. I have also linked to an article (bottom of the post) where the actual experience with faster crawl rate is described.

By: Eric

Eric — Mon, 26 Jan 2009 15:58:43 +0000

What are the pros and cons for a faster crawl rate? I mean if slows your server down that seems like its not worth increasing the crawl rate!

By: paisley

paisley — Mon, 26 Jan 2009 03:03:57 +0000

what if…. there was more than one bot.. one for webmaster tools and one (several – i.e. cloud), that has been running and indexing sites all along which is how google got the largest index footprint in the first place??