Regular and frequent visits by the crawler is the first sign that your site appeals to Google. Thus the most efficient way to get frequent and deep crawls is to develop a website that search engines see as important and valuable.
Note that you can’t force Googlebot to visit you more often – what you can do is to invite it to come. Possible measures to take to increase the crawl rate may include:
- Update your content often and regularly (and ping Google once you do) – well, an obvious one, so not much to describe here; in a word, try to add new unique content as often as you can afford and do it regularly (3 times a week can be the best solution if you can’t update your site daily and are looking for the optimal update rate).
- Make sure your server works correctly: mind the uptime and Google Webmaster tools reports of the unreached pages. Two tools I can recommend here are Pingdom and Mon.itor.us.
- Mind your page load time: note that the crawl works on a budget – if it spends too much time crawling your huge images or PDFs, there will be no time left to visit your other pages.
- Check the site internal link structure: make sure there is no duplicate content returned via different URLs: again, the more time the crawler spends figuring your duplicate content, the fewer useful and unique pages it will manage to visit.
- Get more back links from regularly crawled sites.
- Adjust the crawl speed via Google Webmaster tools.
- Add a sitemap (though it’s up for a debate whether the sitemap can help with crawling and indexing issues, many webmasters report they have seen increased crawl rate after adding it).
- Make sure your server returns the correct header response. Does it handle your error pages properly? Don’t make the bot figure out what has happened: explain it clearly.
- Make sure you have unique title and meta tags for each of your pages.
- Monitor Google crawl rate for your site and see what works and what not:
- access crawl stats via Google Webmaster tools:
- Take advantage of this great WordPress Plugin that tracks crawl rate for Google, Yahoo and MSN:
Hi Ann, Thanks for the informative seo articles…….the site is very useful for beginner bloggers. Today i have read all most every article about crawl and indexation of google. But i did not find my answer. My problem is as follows” I have above 2000 pages website named sarkarinaukri.net. Before 10 Feb 2010 the post title shown in google while search keyword pharse but from that that my post title not shown in google, only blog title is coming. I think something is going to be wrong to my blog.”
In my sitemap shown in google webmaster tool url are also decreasing day by dya
Please suggest
Thanks
Thanks Ann,
Really this is great post and I have question……..
When i am going to build inbound links and how can we checked the crawl rate of a website?
Great post. I have never thought of crawl rate. thank you so much for sharing this kind of information.
Thanks for the info about checking crawl stats in Google's Webmaster Tools. I found that Google is crawling on average 35 pages of mine per day, yet when I checked Google's cached versions of my webpages, it's still displaying pages that are week old. I was hoping there's a way for Google to re-cache my pages more frequently. Any tips?
I want to know, what value we should set to make Google reindex our website faster ? This is make me confuse.
Excellent post thanks Ann!
useful information, but one thing i would like to share in this post, update your post by comment regularly by users, this will bring your page at top level in search engine
1. Submit Articles
2. Make A Blog, related to your site
3. Take regular parts in Forums,
4. Do Submit Some Press Release
5. Comment On Others' Blog, but do not spam them.
6. Do Some Link Building with some same sites, ask webmasters of those sites to do it for you.
7. Do some Directory Submission.
8. Take part in Social Networking sites regularly, like Twitter, facebook, myspace etc.
This can increase your backlinks.
@Ann Smarty
Thanks for you valuable post. I have one question regarding duplicate content in website. I am working on eCommerce website. There are 1000 + SKU & products are repeated in many categories. Will it decrease my crawling rate or not?
I totally agree.. you explanation is correct
I use this for my own websites, and the advice for making backlinks make a big diffrence. :-)
I think these are the basic ways or method to increase the crawl rate. No doubt, website content is king and only useful and quality content. Regular updating in content of website can also increased the chanced of fast crawl.
Thanks for the tips…
I use this for my own websites, and the advice for making backlinks make a big diffrence. :-)
I think these are the basic ways or method to increase the crawl rate. No doubt, website content is king and only useful and quality content. Regular updating in content of website can also increased the chanced of fast crawl.
Thanks for the tips…