Getting Google Crawl Without Indexing?

SMS Text

Very often we have to deal with site (partial) duplicate content issues created by the site CMS, like:



Blocking the pages via Robots.txt Disallow directive will prevent bots from crawling the page but is it probably worth trying to let them spidering the page without indexing them? This may help a lot for discovering more inner pages.

The possible ways to do that (especially now that we don’t have to worry much about PR leakage):

  • Adding “Noindex” robots meta tag to all pages except the first / base one;
  • Adding in your robots.txt the Noindex directive (which is unofficially supported by Google)

User-agent: Googlebot
Noindex: /pricing-high-low/

  • Using rel=canonical (which is the mildest of the three: it won’t prevent Google from crawling but at least it will show Google which pages are more powerful).

So have you ever tried preventing Google from indexing pages without blocking it from spidering them? Does it solve the duplicate content issue anyway?

Ann Smarty
Ann Smarty is the blogger and community manager at Internet Marketing Ninjas. Ann's expertise in blogging and tools serve as a base for her writing, tutorials and her guest blogging project,
Ann Smarty
Get the latest news from Search Engine Journal!
We value your privacy! See our policy here.
  • seobag

    I think that Meta NoIndex much better, because it’s supporting by many search bots.

  • seobag

    I use this code:


    If you have some external links to blocking pages this code don’t block link juice and blocked page can pass link juice to other pages.

  • Alan Bleiweiss


    You always come up with great tech insights and articles – personally I’m doing a test right now on a recently launched site to see which method works best. I just started this week. I’m curious to see if the various methods have different effects.

    Then again, by using different methods on the same site I wonder if that’s going to artificially have an effect on the results.

  • Administrator

    Thanks i read your article very nice article, please write how many types we increase site traffic, please.

    thanks again


  • International Dating Service

    Good to know info. Thanks for the great stuff.


    Thank you Ann, for the article.

  • Payday money

    I can see such doubts constantly raised in Google webmaster forums. People seldom comprehend the nuances in noindex and disallow, which is really a beauty. This would help immensely all the webmasters around the world.

  • Samster

    Thank you.. Ann Smarty for such a important topic or article to share..

  • jeff breeden

    Hi Ann Smarty, I read you article but it is not sufficient about google page crawling without index. So i request you please write more about google crawling and robots.txt.