SEO

10 Ways to Increase Your Site Crawl Rate

Regular and frequent visits by the crawler is the first sign that your site appeals to Google. Thus the most efficient way to get frequent and deep crawls is to develop a website that search engines see as important and valuable.

Note that you can’t force Googlebot to visit you more often – what you can do is to invite it to come. Possible measures to take to increase the crawl rate may include:

  1. Update your content often and regularly (and ping Google once you do) – well, an obvious one, so not much to describe here; in a word, try to add new unique content as often as you can afford and do it regularly (3 times a week can be the best solution if you can’t update your site daily and are looking for the optimal update rate).
  2. Make sure your server works correctly: mind the uptime and Google Webmaster tools reports of the unreached pages. Two tools I can recommend here are Pingdom and Mon.itor.us.
  3. Mind your page load time: note that the crawl works on a budget – if it spends too much time crawling your huge images or PDFs, there will be no time left to visit your other pages.
  4. Check the site internal link structure: make sure there is no duplicate content returned via different URLs: again, the more time the crawler spends figuring your duplicate content, the fewer useful and unique pages it will manage to visit.
  5. Get more back links from regularly crawled sites.
  6. Adjust the crawl speed via Google Webmaster tools.
  7. Add a sitemap (though it’s up for a debate whether the sitemap can help with crawling and indexing issues, many webmasters report they have seen increased crawl rate after adding it).
  8. Make sure your server returns the correct header response. Does it handle your error pages properly? Don’t make the bot figure out what has happened: explain it clearly.
  9. Make sure you have unique title and meta tags for each of your pages.
  10. Monitor Google crawl rate for your site and see what works and what not:

crawl rate google tools 10 Ways to Increase Your Site Crawl Rate

crawl rate wp plugin 10 Ways to Increase Your Site Crawl Rate

 10 Ways to Increase Your Site Crawl Rate
Ann Smarty is the blogger and community manager at Internet Marketing Ninjas. Ann's expertise in blogging and tools serve as a base for her writing, tutorials and her guest blogging project, MyBlogGuest.com.
 10 Ways to Increase Your Site Crawl Rate

You Might Also Like

Comments are closed.

88 thoughts on “10 Ways to Increase Your Site Crawl Rate

  1. Ann,

    Have a small clarification in your last point. If we use plugins to track the crawl(Google) rate, doesn’t mean that we are querying Big G thro third party tool, and its against their guidelines?

    At the same time i accept that the plugin you have mentioned is a real good wonder. I have read reviews on many blogs abt this crawlrate plugin.

  2. The download link for the crawl rate tracker is plugin is broken. Don’t bother trying to leave a comment to let Patrick know because the comment plugin is broken too.

  3. “3 times a week can be the best solution if you can’t update your site daily and are looking for the optimal update rate”

    You getting this number based on some sort of evidence?

  4. @Ann – thanks for letting us know about the great plugin, and good article BTW!

    @Loren – wow, that was a pretty quick response!

    @Patrick – thanks for the update. I’ll definitely go back later and try out that plugin. Sounds pretty neat and thanks for sharing!

  5. @Ann – not really, I only randomly take a look at it for significant dips. In my opinion, they would probably take a look at historical data (no guess as to how far back) to determine crawl rate. If you’ve been updating multiple times per day for the last year they’re going to crawl you more, if you’ve been updating once a week for the last year rate will be much slower. I’d guess you could pretty much control your crawl rate by your update rate and it wouldn’t really be a “certain number of days” thing.

  6. @Ann,

    Lets say I have a popular website which was not updated for more than 3 months. Will increasing only the back-links help in good position in SERPs (especially in google)

    -Pratheep

  7. I am somewhat new to this web site stuff when you say “ping” google in your article are you refering to resubmitting the web pages through something like “web ceo” or other software?

  8. As a niche marketer, we can certainly attest to a lot of the points you make in this article. Now I’ll have to try some of your suggestions that are new to me.

  9. Hmm…. I was getting PR3 prediction for past 4 months … but still Google has not indexed me… Hope it will… It crawls and caches everything I have posted … But not indexing in the Google directory.. STill N/A….

  10. Excellent post, bookmarked! and the deep links and xhtml/css verification is are good tips, add those to your post!

  11. The fact that you can’t change the crawl rate doesn’t mean anything bad. Apparantly, Google has found the most suitable rate for your site and doesn’t want you to interfere.

  12. After i set up everything to Google webmaster tool, do i need to write google to index my page.
    I ask this question because i saw my site said “no index” on the dashboard after i set the the sitemap and sign up my site on google webmaster tool.

  13. It’s really such a nice information you posted here About Site Crawl Rate. I really love the way information presented in your post. I have added to you in my social bookmark…..look forward to hear more from you!!!

    Keep Going for next level

  14. Appreciate the information!

    Regarding “1. Update your content often and regularly (and ping Google once you do),” how to *ping* Google?

  15. open the command prompt, and type in “ping google.com” (without the quotes) and smash ENTER.

  16. I have a site that has plenty of back links and this list is growing daily, the content is changing daily yet it has been over a month and still Google has’nt picked it up – its related/relevant only for the christmas season
    http://www.santas-christmas-letters.com
    if Google doesnt pick it up soon – it will have been a wasted effort – very frustrating

  17. Hello, I would like to comment about using sitemaps. As per my experience, I have worked and still working with several b2b websites of over million pages where content is updated frequently too, that sitemaps on google websmater tool is really effective for small websites. For the websites I just mentioned, managing these sitemaps is hack of job. So without sophisticated programming, do not waste time in sitemaps, search engines are made to crawl our websites to keep their databases fresh, they can not afford to take risk of not crawling anyways.

  18. Great article. I will start working on moe back links just as soon as I figure out where to get some good ones. Any ideas? Thanks again.

  19. I posting content daily my website is 56 month old. 40-50 visitors daily. please check my website and tell me the solution.

  20. my google Crawl stats, Pages crawled per day high:88, Average: 10, Low: 0. what ? this is best crawl stats ? Or not , please tell me anyone coz im new on google webmaster tools.

  21. I submitted some pages of my website in delicious, digg. but now i removed those pages from my website because google adsense fired me duplicate content. now these pages fired 404 error. i want to redirect 404 error page to my homepage..can u advise me?
    how can i do this?

  22. Very simple use 301 redirect the old page with new pages. if you would like to increase the crawl rate update your website content on regular basis. Only quality and updated website text or content can help you to increase the chances of your immediate crawl.

    All the best Sonal!

  23. Hey sonal,

    This post is to increase crawl rate to your pages. I suggest you to post such questions in some other forums or yahoo answers. Its quick.
    Well for your 404 problem I suggest you to implement the errordocument instead of redirecting them to home page.
    Because redirecting will not solve the issue, server response code is important.

    You can implement it with help of .htaccess file if u r running site on apache.
    redirect rule is –
    ErrorDocument 404 /error.php

    This code will redirect all 404 requests to error.php and you design the error.php which will have some message like “The page that you are trying to open is not exists”. also provide some navigation there.

    This will help your users and search engines also.

  24. Very useful tips. I have query regarding the “Crawl rate Graph”. The data shown in the webmaster tool is not matching with my web access-log

  25. Thanks for the helpful tips. I have had a few problems with getting google to crawl my webpages recently and your post has really helped me get my website crawled properly again. This is a really helpful website you have alot of very helpful information. Thanks again

  26. Hi Ann, Thanks for the informative seo articles…….the site is very useful for beginner bloggers. Today i have read all most every article about crawl and indexation of google. But i did not find my answer. My problem is as follows” I have above 2000 pages website named sarkarinaukri.net. Before 10 Feb 2010 the post title shown in google while search keyword pharse but from that that my post title not shown in google, only blog title is coming. I think something is going to be wrong to my blog.”

    In my sitemap shown in google webmaster tool url are also decreasing day by dya

    Please suggest
    Thanks

  27. Thanks Ann,
    Really this is great post and I have question……..
    When i am going to build inbound links and how can we checked the crawl rate of a website?

  28. Great post. I have never thought of crawl rate. thank you so much for sharing this kind of information.

  29. Thanks for the info about checking crawl stats in Google's Webmaster Tools. I found that Google is crawling on average 35 pages of mine per day, yet when I checked Google's cached versions of my webpages, it's still displaying pages that are week old. I was hoping there's a way for Google to re-cache my pages more frequently. Any tips?

  30. useful information, but one thing i would like to share in this post, update your post by comment regularly by users, this will bring your page at top level in search engine

  31. 1. Submit Articles

    2. Make A Blog, related to your site

    3. Take regular parts in Forums,

    4. Do Submit Some Press Release

    5. Comment On Others' Blog, but do not spam them.

    6. Do Some Link Building with some same sites, ask webmasters of those sites to do it for you.

    7. Do some Directory Submission.

    8. Take part in Social Networking sites regularly, like Twitter, facebook, myspace etc.

    This can increase your backlinks.

  32. @Ann Smarty

    Thanks for you valuable post. I have one question regarding duplicate content in website. I am working on eCommerce website. There are 1000 + SKU & products are repeated in many categories. Will it decrease my crawling rate or not?

  33. I think these are the basic ways or method to increase the crawl rate. No doubt, website content is king and only useful and quality content. Regular updating in content of website can also increased the chanced of fast crawl.

    Thanks for the tips…

  34. I think these are the basic ways or method to increase the crawl rate. No doubt, website content is king and only useful and quality content. Regular updating in content of website can also increased the chanced of fast crawl.

    Thanks for the tips…

  35. Hi Ann,

    Thanks for mentioning Monitor.Us in your post (I know it was a while ago now!)

    In the meantime a lot of things have changed with the service – with most of the excellent tools now available free of charge. The only difference between http://Monitor.us (the free version) and http://monitis.com is that the free tool is restricted to 2 locations, a 30 minute interval and no historical reporting – everything else is free!

    Perhaps a follow up post with some more explanation of the tools would be useful for your readers?! Happy to help out.

    thanks
    Ben
    Monitor.Us

  36. well i have a question… i have saw a meta tag previously that will change the google crawl rate to 1 day or 2 days. will taht meta tag helps? tq!