10 Ways to Increase Your Site Crawl Rate

SMS Text

Regular and frequent visits by the crawler is the first sign that your site appeals to Google. Thus the most efficient way to get frequent and deep crawls is to develop a website that search engines see as important and valuable.

Note that you can’t force Googlebot to visit you more often – what you can do is to invite it to come. Possible measures to take to increase the crawl rate may include:

  1. Update your content often and regularly (and ping Google once you do) – well, an obvious one, so not much to describe here; in a word, try to add new unique content as often as you can afford and do it regularly (3 times a week can be the best solution if you can’t update your site daily and are looking for the optimal update rate).
  2. Make sure your server works correctly: mind the uptime and Google Webmaster tools reports of the unreached pages. Two tools I can recommend here are Pingdom and Mon.itor.us.
  3. Mind your page load time: note that the crawl works on a budget – if it spends too much time crawling your huge images or PDFs, there will be no time left to visit your other pages.
  4. Check the site internal link structure: make sure there is no duplicate content returned via different URLs: again, the more time the crawler spends figuring your duplicate content, the fewer useful and unique pages it will manage to visit.
  5. Get more back links from regularly crawled sites.
  6. Adjust the crawl speed via Google Webmaster tools.
  7. Add a sitemap (though it’s up for a debate whether the sitemap can help with crawling and indexing issues, many webmasters report they have seen increased crawl rate after adding it).
  8. Make sure your server returns the correct header response. Does it handle your error pages properly? Don’t make the bot figure out what has happened: explain it clearly.
  9. Make sure you have unique title and meta tags for each of your pages.
  10. Monitor Google crawl rate for your site and see what works and what not:

crawl rate via Google Webmaster tools

crawl rate tracker

Ann Smarty

Ann Smarty

Brand amd Community Manager at Internet Marketing Ninjas
Ann Smarty is the blogger and community manager at Internet Marketing Ninjas. Ann's expertise in blogging and tools serve as a base for her writing,... Read Full Bio
Ann Smarty
Get the latest news from Search Engine Journal!
We value your privacy! See our policy here.
  • Ann,

    Have a small clarification in your last point. If we use plugins to track the crawl(Google) rate, doesn’t mean that we are querying Big G thro third party tool, and its against their guidelines?

    At the same time i accept that the plugin you have mentioned is a real good wonder. I have read reviews on many blogs abt this crawlrate plugin.

  • I will need Patrick Altoft to confirm that but I am pretty sure the plugin analyzes the log files; so it in no way queries Google…

  • Thanks for clarifying!

  • The download link for the crawl rate tracker is plugin is broken. Don’t bother trying to leave a comment to let Patrick know because the comment plugin is broken too.

  • @Austin, I’ve contacted Patrick and alerted him of this issue.

  • @Austin I’ve downloaded the plugin some time ago, so wasn’t aware of that. But the plugin itself works fine with me right now…

  • “3 times a week can be the best solution if you can’t update your site daily and are looking for the optimal update rate”

    You getting this number based on some sort of evidence?

  • Sorry, blogstorm is moving hosts today so quite a few things are broken.

    Please give it 24 hours.

  • @Ann – thanks for letting us know about the great plugin, and good article BTW!

    @Loren – wow, that was a pretty quick response!

    @Patrick – thanks for the update. I’ll definitely go back later and try out that plugin. Sounds pretty neat and thanks for sharing!

  • @TheMadHat – just my personal experience. Do you have another opinion?

  • Nicely summarized. It will be interesting!

  • @Ann – not really, I only randomly take a look at it for significant dips. In my opinion, they would probably take a look at historical data (no guess as to how far back) to determine crawl rate. If you’ve been updating multiple times per day for the last year they’re going to crawl you more, if you’ve been updating once a week for the last year rate will be much slower. I’d guess you could pretty much control your crawl rate by your update rate and it wouldn’t really be a “certain number of days” thing.

  • @Ann,

    Lets say I have a popular website which was not updated for more than 3 months. Will increasing only the back-links help in good position in SERPs (especially in google)


  • Great top 10 points…another one might be making sure your site XHTML and CSS validates.

  • These are excellent tips to better ones crawal rate………I will certainly use this advice………

  • Excellent tips.It´s important to get the site frequently crawled at least if you post a new blogpost every day.

  • Nicely summarized. The crawl rate as I understand from web master tools can only be slowed down if needed, not increased.

  • I am somewhat new to this web site stuff when you say “ping” google in your article are you refering to resubmitting the web pages through something like “web ceo” or other software?

  • As a niche marketer, we can certainly attest to a lot of the points you make in this article. Now I’ll have to try some of your suggestions that are new to me.

  • If you have a big website: Don’t forget about deep links!

  • Hmm…. I was getting PR3 prediction for past 4 months … but still Google has not indexed me… Hope it will… It crawls and caches everything I have posted … But not indexing in the Google directory.. STill N/A….

  • Fantastic ! My blog is crawled every day even if i forget to post some day. I don’t know how google is working

  • very informative posts and with great resource which is really great for my site. thanks

  • Excellent post, bookmarked! and the deep links and xhtml/css verification is are good tips, add those to your post!

  • Nice post for information

  • Thank you for sharing information. The WordPress plugin is really great. So cute graphics. 🙂 It is interesting if it will support more search engines…

  • good one!

  • These are excellent tips for your site crawl…….I have used them many times.

  • Thanks A Smarty for useful tips.

  • Excellent tips for better crawl rate, and I think tips no 5: Get more back links from regularly crawled sites is really work for me

  • Off course, back links from relevant web site and high ranking is most important for get good page rank from Google.


  • How to increase this rate for blogger

  • I was just in my Google Webmaster tools and my most recent sites all say “crawl rate cannot be changed”

  • The fact that you can’t change the crawl rate doesn’t mean anything bad. Apparantly, Google has found the most suitable rate for your site and doesn’t want you to interfere.

  • Thanks for the info dear…really appreciate that


  • Hi pics,
    Please review your webblog site in detail and followed the above excellent tips. I hope you can get yours desire results and your web site crawled successfully.

  • thanks for the informat

  • i found it very usful


  • your information was very good

    Mentor Securio

  • Thanks for this great information. I use this for my own websites, and the advice for making backlinks make a big diffrence. 🙂

  • Mikayel

    for uptime you can check also premium version of mon.itor.us – http://www.monitis.com

  • Good post and the only clarification is how do we increase the crawl rate.

  • Thanks for the great tips !

  • nice information

  • I’m loving the charts 😀

  • It is feasibly impossible to set a crawl rate when using Google’s Blogger.

  • Is backlink the same as deep link?

    Example of a deep link:

  • After i set up everything to Google webmaster tool, do i need to write google to index my page.
    I ask this question because i saw my site said “no index” on the dashboard after i set the the sitemap and sign up my site on google webmaster tool.

    • I am sure Google will figure that once you remove noindex… Just give it some time.

  • Thanks for the tips !

  • It’s really such a nice information you posted here About Site Crawl Rate. I really love the way information presented in your post. I have added to you in my social bookmark…..look forward to hear more from you!!!

    Keep Going for next level

  • Appreciate the information!

    Regarding “1. Update your content often and regularly (and ping Google once you do),” how to *ping* Google?

  • nadrolski

    open the command prompt, and type in “ping google.com” (without the quotes) and smash ENTER.

  • What I would like to know is what the sites that Google crawls often?

  • the “ping google” is not clear for me… How does it work?

  • I have a site that has plenty of back links and this list is growing daily, the content is changing daily yet it has been over a month and still Google has’nt picked it up – its related/relevant only for the christmas season
    if Google doesnt pick it up soon – it will have been a wasted effort – very frustrating

  • Hello, I would like to comment about using sitemaps. As per my experience, I have worked and still working with several b2b websites of over million pages where content is updated frequently too, that sitemaps on google websmater tool is really effective for small websites. For the websites I just mentioned, managing these sitemaps is hack of job. So without sophisticated programming, do not waste time in sitemaps, search engines are made to crawl our websites to keep their databases fresh, they can not afford to take risk of not crawling anyways.

  • Blaine sword

    Great article. I will start working on moe back links just as soon as I figure out where to get some good ones. Any ideas? Thanks again.

  • I posting content daily my website is 56 month old. 40-50 visitors daily. please check my website and tell me the solution.

  • my google Crawl stats, Pages crawled per day high:88, Average: 10, Low: 0. what ? this is best crawl stats ? Or not , please tell me anyone coz im new on google webmaster tools.

  • Hey thanks a lot for these steps. I am going to start working around these steps to increase my sites crawl rate.

  • I submitted some pages of my website in delicious, digg. but now i removed those pages from my website because google adsense fired me duplicate content. now these pages fired 404 error. i want to redirect 404 error page to my homepage..can u advise me?
    how can i do this?

  • Very simple use 301 redirect the old page with new pages. if you would like to increase the crawl rate update your website content on regular basis. Only quality and updated website text or content can help you to increase the chances of your immediate crawl.

    All the best Sonal!

  • Hey sonal,

    This post is to increase crawl rate to your pages. I suggest you to post such questions in some other forums or yahoo answers. Its quick.
    Well for your 404 problem I suggest you to implement the errordocument instead of redirecting them to home page.
    Because redirecting will not solve the issue, server response code is important.

    You can implement it with help of .htaccess file if u r running site on apache.
    redirect rule is –
    ErrorDocument 404 /error.php

    This code will redirect all 404 requests to error.php and you design the error.php which will have some message like “The page that you are trying to open is not exists”. also provide some navigation there.

    This will help your users and search engines also.

  • Very useful tips. I have query regarding the “Crawl rate Graph”. The data shown in the webmaster tool is not matching with my web access-log

  • Thanks for the helpful tips. I have had a few problems with getting google to crawl my webpages recently and your post has really helped me get my website crawled properly again. This is a really helpful website you have alot of very helpful information. Thanks again

  • Hi Ann, Thanks for the informative seo articles…….the site is very useful for beginner bloggers. Today i have read all most every article about crawl and indexation of google. But i did not find my answer. My problem is as follows” I have above 2000 pages website named sarkarinaukri.net. Before 10 Feb 2010 the post title shown in google while search keyword pharse but from that that my post title not shown in google, only blog title is coming. I think something is going to be wrong to my blog.”

    In my sitemap shown in google webmaster tool url are also decreasing day by dya

    Please suggest

  • Thanks Ann,
    Really this is great post and I have question……..
    When i am going to build inbound links and how can we checked the crawl rate of a website?

  • chelseybuhler

    Great post. I have never thought of crawl rate. thank you so much for sharing this kind of information.

  • Thanks for the info about checking crawl stats in Google's Webmaster Tools. I found that Google is crawling on average 35 pages of mine per day, yet when I checked Google's cached versions of my webpages, it's still displaying pages that are week old. I was hoping there's a way for Google to re-cache my pages more frequently. Any tips?

  • I want to know, what value we should set to make Google reindex our website faster ? This is make me confuse.

  • Excellent post thanks Ann!

  • useful information, but one thing i would like to share in this post, update your post by comment regularly by users, this will bring your page at top level in search engine

  • marckdon

    1. Submit Articles

    2. Make A Blog, related to your site

    3. Take regular parts in Forums,

    4. Do Submit Some Press Release

    5. Comment On Others' Blog, but do not spam them.

    6. Do Some Link Building with some same sites, ask webmasters of those sites to do it for you.

    7. Do some Directory Submission.

    8. Take part in Social Networking sites regularly, like Twitter, facebook, myspace etc.

    This can increase your backlinks.

  • anandmistry

    @Ann Smarty

    Thanks for you valuable post. I have one question regarding duplicate content in website. I am working on eCommerce website. There are 1000 + SKU & products are repeated in many categories. Will it decrease my crawling rate or not?

  • I totally agree.. you explanation is correct

  • I use this for my own websites, and the advice for making backlinks make a big diffrence. 🙂

  • I think these are the basic ways or method to increase the crawl rate. No doubt, website content is king and only useful and quality content. Regular updating in content of website can also increased the chanced of fast crawl.

    Thanks for the tips…

  • I use this for my own websites, and the advice for making backlinks make a big diffrence. 🙂

  • I think these are the basic ways or method to increase the crawl rate. No doubt, website content is king and only useful and quality content. Regular updating in content of website can also increased the chanced of fast crawl.

    Thanks for the tips…

  • Hi Ann,

    Thanks for mentioning Monitor.Us in your post (I know it was a while ago now!)

    In the meantime a lot of things have changed with the service – with most of the excellent tools now available free of charge. The only difference between http://Monitor.us (the free version) and http://monitis.com is that the free tool is restricted to 2 locations, a 30 minute interval and no historical reporting – everything else is free!

    Perhaps a follow up post with some more explanation of the tools would be useful for your readers?! Happy to help out.


  • I’m trying to get some more traffic to my websites, so this is good advice. Thanks.

  • bilik sewa

    well i have a question… i have saw a meta tag previously that will change the google crawl rate to 1 day or 2 days. will taht meta tag helps? tq!

  • Hi Thanks for the tips , hope soon will get some changes in indexing of my site

  • Mik

    thank you

  • yeah this is a great peace of info thnx alot helped me alot..

  • Thanks for the tips. Didn’t publish articles for a month, and now I’m struggling to get google to index new pages.

  • Wow! sitemap thing works really well. and thanks for sharing Pingdom.