10 Ways to Increase Your Site Crawl Rate

SMS Text

Regular and frequent visits by the crawler is the first sign that your site appeals to Google. Thus the most efficient way to get frequent and deep crawls is to develop a website that search engines see as important and valuable.

Note that you can’t force Googlebot to visit you more often – what you can do is to invite it to come. Possible measures to take to increase the crawl rate may include:

  1. Update your content often and regularly (and ping Google once you do) – well, an obvious one, so not much to describe here; in a word, try to add new unique content as often as you can afford and do it regularly (3 times a week can be the best solution if you can’t update your site daily and are looking for the optimal update rate).
  2. Make sure your server works correctly: mind the uptime and Google Webmaster tools reports of the unreached pages. Two tools I can recommend here are Pingdom and Mon.itor.us.
  3. Mind your page load time: note that the crawl works on a budget – if it spends too much time crawling your huge images or PDFs, there will be no time left to visit your other pages.
  4. Check the site internal link structure: make sure there is no duplicate content returned via different URLs: again, the more time the crawler spends figuring your duplicate content, the fewer useful and unique pages it will manage to visit.
  5. Get more back links from regularly crawled sites.
  6. Adjust the crawl speed via Google Webmaster tools.
  7. Add a sitemap (though it’s up for a debate whether the sitemap can help with crawling and indexing issues, many webmasters report they have seen increased crawl rate after adding it).
  8. Make sure your server returns the correct header response. Does it handle your error pages properly? Don’t make the bot figure out what has happened: explain it clearly.
  9. Make sure you have unique title and meta tags for each of your pages.
  10. Monitor Google crawl rate for your site and see what works and what not:

crawl rate via Google Webmaster tools

crawl rate tracker

Ann Smarty
Ann Smarty is the blogger and community manager at Internet Marketing Ninjas. Ann's expertise in blogging and tools serve as a base for her writing, tutorials and her guest blogging project, MyBlogGuest.com.
Ann Smarty
  • http://www.dotcominfoway.fr/ Mercy

    Ann,

    Have a small clarification in your last point. If we use plugins to track the crawl(Google) rate, doesn’t mean that we are querying Big G thro third party tool, and its against their guidelines?

    At the same time i accept that the plugin you have mentioned is a real good wonder. I have read reviews on many blogs abt this crawlrate plugin.

  • http://www.seosmarty.com/ Ann Smarty

    I will need Patrick Altoft to confirm that but I am pretty sure the plugin analyzes the log files; so it in no way queries Google…

  • http://www.dotcominfoway.fr/ Mercy

    Thanks for clarifying!

  • http://www.northofthesavannah.com Austin

    The download link for the crawl rate tracker is plugin is broken. Don’t bother trying to leave a comment to let Patrick know because the comment plugin is broken too.

  • https://www.searchenginejournal.com Loren Baker, Editor

    @Austin, I’ve contacted Patrick and alerted him of this issue.

  • http://www.seosmarty.com/ Ann Smarty

    @Austin I’ve downloaded the plugin some time ago, so wasn’t aware of that. But the plugin itself works fine with me right now…

  • http://www.themadhat.com TheMadHat

    “3 times a week can be the best solution if you can’t update your site daily and are looking for the optimal update rate”

    You getting this number based on some sort of evidence?

  • http://blogstorm.co.uk Patrick Altoft

    Sorry, blogstorm is moving hosts today so quite a few things are broken.

    Please give it 24 hours.

  • http://www.northofthesavannah.com Austin

    @Ann – thanks for letting us know about the great plugin, and good article BTW!

    @Loren – wow, that was a pretty quick response!

    @Patrick – thanks for the update. I’ll definitely go back later and try out that plugin. Sounds pretty neat and thanks for sharing!

  • http://www.seosmarty.com/ Ann Smarty

    @TheMadHat – just my personal experience. Do you have another opinion?

  • http://www.realtor.com.ru Realtor

    Nicely summarized. It will be interesting!

  • http://www.themadhat.com TheMadHat

    @Ann – not really, I only randomly take a look at it for significant dips. In my opinion, they would probably take a look at historical data (no guess as to how far back) to determine crawl rate. If you’ve been updating multiple times per day for the last year they’re going to crawl you more, if you’ve been updating once a week for the last year rate will be much slower. I’d guess you could pretty much control your crawl rate by your update rate and it wouldn’t really be a “certain number of days” thing.

  • http://matt-cutts.blogspot.com/ Pratheep

    @Ann,

    Lets say I have a popular website which was not updated for more than 3 months. Will increasing only the back-links help in good position in SERPs (especially in google)

    -Pratheep

  • http://www.searchengineacademysc.com Tracy Fredrychowski

    Great top 10 points…another one might be making sure your site XHTML and CSS validates.

  • http://www.vectorpedia.com Vectorpedia(Rick)

    These are excellent tips to better ones crawal rate………I will certainly use this advice………

  • http://tlmarketing.net Tom At The Home Business Archive

    Excellent tips.It´s important to get the site frequently crawled at least if you post a new blogpost every day.

  • http://articlewonders.com Raj Krishnaswamy

    Nicely summarized. The crawl rate as I understand from web master tools can only be slowed down if needed, not increased.

  • http://www.thebacheloretteboutique.comandwww.gasparstreasures.com Bill

    I am somewhat new to this web site stuff when you say “ping” google in your article are you refering to resubmitting the web pages through something like “web ceo” or other software?

  • http://www.abncparties.com Michael

    As a niche marketer, we can certainly attest to a lot of the points you make in this article. Now I’ll have to try some of your suggestions that are new to me.

  • http://www.lorm.de/ Malte Landwehr

    If you have a big website: Don’t forget about deep links!

  • http://sunilmblog.blogspot.com Sunil

    Hmm…. I was getting PR3 prediction for past 4 months … but still Google has not indexed me… Hope it will… It crawls and caches everything I have posted … But not indexing in the Google directory.. STill N/A….

  • http://www.blueunplugged.com/p.aspx?p=121764 Nintendo wii

    Fantastic ! My blog is crawled every day even if i forget to post some day. I don’t know how google is working

  • http://www.opengiga.com OPENGIGA

    very informative posts and with great resource which is really great for my site. thanks

  • http://www.blogodirectory.com Mel

    Excellent post, bookmarked! and the deep links and xhtml/css verification is are good tips, add those to your post!

  • http://www.info-gadget.co.cc Heri

    Nice post for information

  • http://mygoodlamp.net/ Aleks@Modern Lamps

    Thank you for sharing information. The WordPress plugin is really great. So cute graphics. :) It is interesting if it will support more search engines…

  • http://www.lehren.tv deepsi

    good one!

  • http://www.turbochargedlinking.com SEO Links

    These are excellent tips for your site crawl…….I have used them many times.

  • http://www.searchbiddrive.com SearchBidDrive

    Thanks A Smarty for useful tips.

  • http://www.baliinc.net/ Baliinc

    Excellent tips for better crawl rate, and I think tips no 5: Get more back links from regularly crawled sites is really work for me

  • http://www.edgetechnologies.co.uk Edgetechnologies UK

    Off course, back links from relevant web site and high ranking is most important for get good page rank from Google.

    Safdar

  • http://picsjunk.blogspot.com Pics

    How to increase this rate for blogger

  • http://jobsboston.info Jobs Boston

    I was just in my Google Webmaster tools and my most recent sites all say “crawl rate cannot be changed”

  • http://www.seosmarty.com/ Ann Smarty

    The fact that you can’t change the crawl rate doesn’t mean anything bad. Apparantly, Google has found the most suitable rate for your site and doesn’t want you to interfere.

  • http://www.vickyadvani.blog.co.in vicky

    Thanks for the info dear…really appreciate that

    Vicky
    http://www.vickyadvani.blog.co.in

  • http://www.hashe.com Hashe Computer Solutions

    Hi pics,
    Please review your webblog site in detail and followed the above excellent tips. I hope you can get yours desire results and your web site crawled successfully.

  • http://www.lehren.tv deepsiion

    thanks for the informat

  • http://www.mentorsecurio.com victoria Eliza

    i found it very usful

    vicky

  • http://www.mentorsecurio.com victoria Eliza

    your information was very good

    Mentor Securio

  • http://www.gratisspil.eu/spil/Puslespil Puslespil

    Thanks for this great information. I use this for my own websites, and the advice for making backlinks make a big diffrence. :-)

  • Mikayel

    for uptime you can check also premium version of mon.itor.us – http://www.monitis.com

  • http://www.bookadda.com Dean

    Good post and the only clarification is how do we increase the crawl rate.

  • http://www.shoptimesaver.com michael

    Thanks for the great tips !

  • http://fastonlineearning.blogspot.com shahid mahmood

    nice information

  • http://www.whatload.com Tom Allen

    I’m loving the charts 😀

  • http://www.choicehow.com Mr.Choice

    It is feasibly impossible to set a crawl rate when using Google’s Blogger.

  • http://www.nwvd.org bsdinfo

    Is backlink the same as deep link?

    Example of a deep link:
    http://www.nwvd.org/cisco/cisco_lab/index.php

  • http://searchclickweb.com leo

    After i set up everything to Google webmaster tool, do i need to write google to index my page.
    I ask this question because i saw my site said “no index” on the dashboard after i set the the sitemap and sign up my site on google webmaster tool.

    • https://www.searchenginejournal.com Ann Smarty

      I am sure Google will figure that once you remove noindex… Just give it some time.

  • http://www.cybervationinc.com Debbie Ray

    Thanks for the tips !

  • http://www.ultimateairplanes.com Cessna Aircrafts

    It’s really such a nice information you posted here About Site Crawl Rate. I really love the way information presented in your post. I have added to you in my social bookmark…..look forward to hear more from you!!!

    Keep Going for next level

  • http://www.bizdesign.org biz

    Appreciate the information!

    Regarding “1. Update your content often and regularly (and ping Google once you do),” how to *ping* Google?

  • nadrolski

    open the command prompt, and type in “ping google.com” (without the quotes) and smash ENTER.

  • http://palmistryforyou.com/ Nita

    What I would like to know is what the sites that Google crawls often?

  • http://talks.110mb.com tks

    the “ping google” is not clear for me… How does it work?

  • http://www.santas-christmas-letters.com kev dock

    I have a site that has plenty of back links and this list is growing daily, the content is changing daily yet it has been over a month and still Google has’nt picked it up – its related/relevant only for the christmas season
    http://www.santas-christmas-letters.com
    if Google doesnt pick it up soon – it will have been a wasted effort – very frustrating

  • http://b2b-trade-international.blogspot.com B2B Trade

    Hello, I would like to comment about using sitemaps. As per my experience, I have worked and still working with several b2b websites of over million pages where content is updated frequently too, that sitemaps on google websmater tool is really effective for small websites. For the websites I just mentioned, managing these sitemaps is hack of job. So without sophisticated programming, do not waste time in sitemaps, search engines are made to crawl our websites to keep their databases fresh, they can not afford to take risk of not crawling anyways.

  • Blaine sword

    Great article. I will start working on moe back links just as soon as I figure out where to get some good ones. Any ideas? Thanks again.

  • http://www.jobs89.com amit

    I posting content daily my website is 56 month old. 40-50 visitors daily. please check my website and tell me the solution.

  • http://www.mysmsbank.com Asad

    my google Crawl stats, Pages crawled per day high:88, Average: 10, Low: 0. what ? this is best crawl stats ? Or not , please tell me anyone coz im new on google webmaster tools.

  • http://twitter.com/dhaneshmane dhaneshmane

    Hey thanks a lot for these steps. I am going to start working around these steps to increase my sites crawl rate.

  • http://www.codventure.com sonal

    I submitted some pages of my website in delicious, digg. but now i removed those pages from my website because google adsense fired me duplicate content. now these pages fired 404 error. i want to redirect 404 error page to my homepage..can u advise me?
    how can i do this?

  • http://www.chauffeurlinks.com Airport Transfer

    Very simple use 301 redirect the old page with new pages. if you would like to increase the crawl rate update your website content on regular basis. Only quality and updated website text or content can help you to increase the chances of your immediate crawl.

    All the best Sonal!

  • http://dhanesh-mane.blogspot.com/ dhanesh mane

    Hey sonal,

    This post is to increase crawl rate to your pages. I suggest you to post such questions in some other forums or yahoo answers. Its quick.
    Well for your 404 problem I suggest you to implement the errordocument instead of redirecting them to home page.
    Because redirecting will not solve the issue, server response code is important.

    You can implement it with help of .htaccess file if u r running site on apache.
    redirect rule is –
    ErrorDocument 404 /error.php

    This code will redirect all 404 requests to error.php and you design the error.php which will have some message like “The page that you are trying to open is not exists”. also provide some navigation there.

    This will help your users and search engines also.

  • http://talks.110mb.com gptc team

    Very useful tips. I have query regarding the “Crawl rate Graph”. The data shown in the webmaster tool is not matching with my web access-log

  • http://www.about-muscle.com james

    Thanks for the helpful tips. I have had a few problems with getting google to crawl my webpages recently and your post has really helped me get my website crawled properly again. This is a really helpful website you have alot of very helpful information. Thanks again

  • http://www.sarkarinaukri.net Pankaj Singla

    Hi Ann, Thanks for the informative seo articles…….the site is very useful for beginner bloggers. Today i have read all most every article about crawl and indexation of google. But i did not find my answer. My problem is as follows” I have above 2000 pages website named sarkarinaukri.net. Before 10 Feb 2010 the post title shown in google while search keyword pharse but from that that my post title not shown in google, only blog title is coming. I think something is going to be wrong to my blog.”

    In my sitemap shown in google webmaster tool url are also decreasing day by dya

    Please suggest
    Thanks

  • http://www.krabivilla.com Krabi

    Thanks Ann,
    Really this is great post and I have question……..
    When i am going to build inbound links and how can we checked the crawl rate of a website?

  • chelseybuhler

    Great post. I have never thought of crawl rate. thank you so much for sharing this kind of information.

  • http://www.zadling.com/ Zadling

    Thanks for the info about checking crawl stats in Google's Webmaster Tools. I found that Google is crawling on average 35 pages of mine per day, yet when I checked Google's cached versions of my webpages, it's still displaying pages that are week old. I was hoping there's a way for Google to re-cache my pages more frequently. Any tips?

  • http://www.massiliabalivilla.com putude

    I want to know, what value we should set to make Google reindex our website faster ? This is make me confuse.

  • http://www.onlinewill.co.uk Wills Online

    Excellent post thanks Ann!

  • http://www.pondicherrytimes.com pondicherry times

    useful information, but one thing i would like to share in this post, update your post by comment regularly by users, this will bring your page at top level in search engine

  • marckdon

    1. Submit Articles

    2. Make A Blog, related to your site

    3. Take regular parts in Forums,

    4. Do Submit Some Press Release

    5. Comment On Others' Blog, but do not spam them.

    6. Do Some Link Building with some same sites, ask webmasters of those sites to do it for you.

    7. Do some Directory Submission.

    8. Take part in Social Networking sites regularly, like Twitter, facebook, myspace etc.

    This can increase your backlinks.

  • anandmistry

    @Ann Smarty

    Thanks for you valuable post. I have one question regarding duplicate content in website. I am working on eCommerce website. There are 1000 + SKU & products are repeated in many categories. Will it decrease my crawling rate or not?

  • http://www.gardendezign.com Weber

    I totally agree.. you explanation is correct

  • http://www.newyorkbarstore.com/oak-bar-stools oak bar stool

    I use this for my own websites, and the advice for making backlinks make a big diffrence. :-)

  • http://www.dacou.com.au Gloria Petyarre

    I think these are the basic ways or method to increase the crawl rate. No doubt, website content is king and only useful and quality content. Regular updating in content of website can also increased the chanced of fast crawl.

    Thanks for the tips…

  • http://www.newyorkbarstore.com/oak-bar-stools oak bar stool

    I use this for my own websites, and the advice for making backlinks make a big diffrence. :-)

  • http://www.dacou.com.au Gloria Petyarre

    I think these are the basic ways or method to increase the crawl rate. No doubt, website content is king and only useful and quality content. Regular updating in content of website can also increased the chanced of fast crawl.

    Thanks for the tips…

  • http://www.monitor.us Ben Rogers

    Hi Ann,

    Thanks for mentioning Monitor.Us in your post (I know it was a while ago now!)

    In the meantime a lot of things have changed with the service – with most of the excellent tools now available free of charge. The only difference between http://Monitor.us (the free version) and http://monitis.com is that the free tool is restricted to 2 locations, a 30 minute interval and no historical reporting – everything else is free!

    Perhaps a follow up post with some more explanation of the tools would be useful for your readers?! Happy to help out.

    thanks
    Ben
    Monitor.Us

  • http://www.readysteadymagic.co.uk Ed

    I’m trying to get some more traffic to my websites, so this is good advice. Thanks.

  • bilik sewa

    well i have a question… i have saw a meta tag previously that will change the google crawl rate to 1 day or 2 days. will taht meta tag helps? tq!

  • http://speakjournal.com Arun rana

    Hi Thanks for the tips , hope soon will get some changes in indexing of my site

  • Mik

    thank you

  • http://www.Beingtricky.com Prince Ramgarhia

    yeah this is a great peace of info thnx alot helped me alot..

  • http://www.invitationtemplatesfree.com John

    Thanks for the tips. Didn’t publish articles for a month, and now I’m struggling to get google to index new pages.

  • http://www.lyricsbug.com Shahbaz

    Wow! sitemap thing works really well. and thanks for sharing Pingdom.