Advertisement
|

10 Ways to Increase Your Site Crawl Rate

  • 83K
    READS
Ann Smarty
Ann Smarty VIP CONTRIBUTOR
10 Ways to Increase Your Site Crawl Rate

Regular and frequent visits by the crawler are the first sign that your site appeals to Google. Thus the most efficient way to get frequent and deep crawls is to develop a website that search engines see as important and valuable.

Note that you can’t force Googlebot to visit you more often – what you can do is to invite it to come. Possible measures to take to increase the crawl rate may include:

Update Your Content Often (and ping Google once you do)

An obvious one, so not much to describe here; in a word, try to add new unique content as often as you can afford and do it regularly (3 times a week can be the best solution if you can’t update your site daily and are looking for the optimal update rate).

Check Your Server

Make sure your server works correctly: mind the uptime and Google Webmaster tools reports of the unreached pages. Two tools I can recommend here are Pingdom and Mon.itor.us.

Pay Attention To Load Time

Mind your page load time: note that the crawl works on a budget – if it spends too much time crawling your huge images or PDFs, there will be no time left to visit your other pages.

Check Links

Check the site internal link structure: make sure there is no duplicate content returned via different URLs: again, the more time the crawler spends figuring your duplicate content, the fewer useful and unique pages it will manage to visit.

Build More Links

Get more back links from regularly crawled sites.

Add a Sitemap

Though it’s up for adebate whether the sitemap can help with crawling and indexing issues, many webmasters report they have seen increased crawl rate after adding it.

Make It Easy

Make sure your server returns the correct header response. Does it handle your error pages properly? Don’t make the bot figure out what has happened: explain it clearly.

Check Meta and Title Tags

Make sure you have unique title and meta tags for each of your pages.

Test, Test, Test

Monitor Google crawl rate for your site and see what works and what not:

Access crawl stats via Google Webmaster tools:

crawl rate via Google Webmaster tools

Try to Get More Social Shares

Although Google maintains that social links are not a ranking factor, since Google does crawl those sites it may help your site be crawled more often.

Final Thoughts

There are multiple benefits of a faster crawl rate. The faster (and more often) Google crawls your site the faster it will find, and hopefully rank, your awesome new content. It can also help you bounce back quicker should you suffer from a penalty.

Any other tips? Share them in the comments section! 

Featured Image: Deposit Photos

 

ADVERTISEMENT
ADVERTISEMENT
Ann Smarty

Ann Smarty

Brand amd Community Manager at Internet Marketing Ninjas

Ann Smarty is the blogger and community manager at Internet Marketing Ninjas. Ann's expertise in blogging and tools serve as ... [Read full bio]

Advertisement