Regular and frequent visits by the crawler is the first sign that your site appeals to Google. Thus the most efficient way to get frequent and deep crawls is to develop a website that search engines see as important and valuable.
Note that you can’t force Googlebot to visit you more often - what you can do is to invite it to come. Possible measures to take to increase the crawl rate may include:
- Update your content often and regularly (and ping Google once you do) - well, an obvious one, so not much to describe here; in a word, try to add new unique content as often as you can afford and do it regularly (3 times a week can be the best solution if you can’t update your site daily and are looking for the optimal update rate).
- Make sure your server works correctly: mind the uptime and Google Webmaster tools reports of the unreached pages. Two tools I can recommend here are Pingdom and Mon.itor.us.
- Mind your page load time: note that the crawl works on a budget - if it spends too much time crawling your huge images or PDFs, there will be no time left to visit your other pages.
- Check the site internal link structure: make sure there is no duplicate content returned via different URLs: again, the more time the crawler spends figuring your duplicate content, the fewer useful and unique pages it will manage to visit.
- Get more back links from regularly crawled sites.
- Adjust the crawl speed via Google Webmaster tools.
- Add a sitemap (though it’s up for a debate whether the sitemap can help with crawling and indexing issues, many webmasters report they have seen increased crawl rate after adding it).
- Make sure your server returns the correct header response. Does it handle your error pages properly? Don’t make the bot figure out what has happened: explain it clearly.
- Make sure you have unique title and meta tags for each of your pages.
- Monitor Google crawl rate for your site and see what works and what not:
- access crawl stats via Google Webmaster tools:
- Take advantage of this great Wordpress Plugin that tracks crawl rate for Google, Yahoo and MSN:
Comments
45 responses so far ↓
Mercy on Jun 24, 2008 at 9:05 am
Ann,
Have a small clarification in your last point. If we use plugins to track the crawl(Google) rate, doesn’t mean that we are querying Big G thro third party tool, and its against their guidelines?
At the same time i accept that the plugin you have mentioned is a real good wonder. I have read reviews on many blogs abt this crawlrate plugin.
Ann Smarty on Jun 24, 2008 at 9:21 am
I will need Patrick Altoft to confirm that but I am pretty sure the plugin analyzes the log files; so it in no way queries Google…
Mercy on Jun 24, 2008 at 9:27 am
Thanks for clarifying!
Austin on Jun 24, 2008 at 9:40 am
The download link for the crawl rate tracker is plugin is broken. Don’t bother trying to leave a comment to let Patrick know because the comment plugin is broken too.
Loren Baker, Editor on Jun 24, 2008 at 9:43 am
@Austin, I’ve contacted Patrick and alerted him of this issue.
Ann Smarty on Jun 24, 2008 at 9:45 am
@Austin I’ve downloaded the plugin some time ago, so wasn’t aware of that. But the plugin itself works fine with me right now…
TheMadHat on Jun 24, 2008 at 9:49 am
“3 times a week can be the best solution if you can’t update your site daily and are looking for the optimal update rate”
You getting this number based on some sort of evidence?
Patrick Altoft on Jun 24, 2008 at 9:51 am
Sorry, blogstorm is moving hosts today so quite a few things are broken.
Please give it 24 hours.
Austin on Jun 24, 2008 at 9:56 am
@Ann - thanks for letting us know about the great plugin, and good article BTW!
@Loren - wow, that was a pretty quick response!
@Patrick - thanks for the update. I’ll definitely go back later and try out that plugin. Sounds pretty neat and thanks for sharing!
Ann Smarty on Jun 24, 2008 at 10:36 am
@TheMadHat - just my personal experience. Do you have another opinion?
Realtor on Jun 24, 2008 at 12:21 pm
Nicely summarized. It will be interesting!
TheMadHat on Jun 24, 2008 at 12:56 pm
@Ann - not really, I only randomly take a look at it for significant dips. In my opinion, they would probably take a look at historical data (no guess as to how far back) to determine crawl rate. If you’ve been updating multiple times per day for the last year they’re going to crawl you more, if you’ve been updating once a week for the last year rate will be much slower. I’d guess you could pretty much control your crawl rate by your update rate and it wouldn’t really be a “certain number of days” thing.
Pratheep on Jun 24, 2008 at 2:19 pm
@Ann,
Lets say I have a popular website which was not updated for more than 3 months. Will increasing only the back-links help in good position in SERPs (especially in google)
-Pratheep
Tracy Fredrychowski on Jun 25, 2008 at 6:30 am
Great top 10 points…another one might be making sure your site XHTML and CSS validates.
Vectorpedia(Rick) on Jun 25, 2008 at 6:41 am
These are excellent tips to better ones crawal rate………I will certainly use this advice………
Tom At The Home Business Archive on Jun 25, 2008 at 12:15 pm
Excellent tips.It´s important to get the site frequently crawled at least if you post a new blogpost every day.
Raj Krishnaswamy on Jun 25, 2008 at 12:51 pm
Nicely summarized. The crawl rate as I understand from web master tools can only be slowed down if needed, not increased.
Bill on Jun 25, 2008 at 5:09 pm
I am somewhat new to this web site stuff when you say “ping” google in your article are you refering to resubmitting the web pages through something like “web ceo” or other software?
Michael on Jun 25, 2008 at 5:55 pm
As a niche marketer, we can certainly attest to a lot of the points you make in this article. Now I’ll have to try some of your suggestions that are new to me.
Malte Landwehr on Jun 26, 2008 at 6:02 am
If you have a big website: Don’t forget about deep links!
Sunil on Jun 29, 2008 at 9:05 am
Hmm…. I was getting PR3 prediction for past 4 months … but still Google has not indexed me… Hope it will… It crawls and caches everything I have posted … But not indexing in the Google directory.. STill N/A….
Nintendo wii on Jul 2, 2008 at 2:06 am
Fantastic ! My blog is crawled every day even if i forget to post some day. I don’t know how google is working
OPENGIGA on Jul 7, 2008 at 6:13 am
very informative posts and with great resource which is really great for my site. thanks
Mel on Aug 23, 2008 at 5:40 pm
Excellent post, bookmarked! and the deep links and xhtml/css verification is are good tips, add those to your post!
Heri on Sep 12, 2008 at 5:22 pm
Nice post for information
Aleks@Modern Lamps on Oct 10, 2008 at 9:23 am
Thank you for sharing information. The Wordpress plugin is really great. So cute graphics. :) It is interesting if it will support more search engines…
deepsi on Oct 15, 2008 at 1:37 pm
good one!
SEO Links on Nov 15, 2008 at 7:31 pm
These are excellent tips for your site crawl…….I have used them many times.
SearchBidDrive on Dec 2, 2008 at 8:54 am
Thanks A Smarty for useful tips.
Baliinc on Dec 13, 2008 at 2:10 am
Excellent tips for better crawl rate, and I think tips no 5: Get more back links from regularly crawled sites is really work for me
Edgetechnologies UK on Jan 1, 2009 at 1:35 pm
Off course, back links from relevant web site and high ranking is most important for get good page rank from Google.
Safdar
Pics on Jan 11, 2009 at 6:30 am
How to increase this rate for blogger
Jobs Boston on Jan 12, 2009 at 4:46 pm
I was just in my Google Webmaster tools and my most recent sites all say “crawl rate cannot be changed”
Ann Smarty on Jan 13, 2009 at 5:09 am
The fact that you can’t change the crawl rate doesn’t mean anything bad. Apparantly, Google has found the most suitable rate for your site and doesn’t want you to interfere.
vicky on Jan 18, 2009 at 2:58 am
Thanks for the info dear…really appreciate that
Vicky
http://www.vickyadvani.blog.co.in
Hashe Computer Solutions on Jan 22, 2009 at 7:22 am
Hi pics,
Please review your webblog site in detail and followed the above excellent tips. I hope you can get yours desire results and your web site crawled successfully.
deepsiion on Feb 6, 2009 at 12:30 am
thanks for the informat
victoria Eliza on Mar 21, 2009 at 2:56 am
i found it very usful
vicky
victoria Eliza on Mar 21, 2009 at 2:59 am
your information was very good
Mentor Securio
Puslespil on Mar 21, 2009 at 8:02 am
Thanks for this great information. I use this for my own websites, and the advice for making backlinks make a big diffrence. :-)
Mikayel on May 10, 2009 at 1:42 pm
for uptime you can check also premium version of mon.itor.us - http://www.monitis.com
Dean on Jun 5, 2009 at 11:52 am
Good post and the only clarification is how do we increase the crawl rate.
michael on Jun 14, 2009 at 1:37 am
Thanks for the great tips !
shahid mahmood on Jun 24, 2009 at 1:16 am
nice information
Tom Allen on Jul 1, 2009 at 12:25 pm
I’m loving the charts :D
Leave a Comment