SEO

No Indexing WordPress Taxonomies: Do or Don’t

One question on the mind of many SEO professionals and bloggers is whether or not to NoIndex archive taxonomies in content management systems. Many content management systems have taxonomies for author archives, date archives, categories, and tags.

WordPress is the most popular CMS and blogging platform in the world. While it is great out-of-the-box, it does come with many flaws. The biggest being the massive amounts of duplicate content it creates.

WordPress tends to create over a half dozen duplicate copies of posts if all taxonomies are used. Since WordPress is used on most company blogs and used to create many company websites, I decided to run an experiment on NoIndexing WP taxonomies.

wordpress taxonomies No Indexing WordPress Taxonomies: Do or Don’t

WordPress Taxonomies

Of course, it would be unethical to try this experiment on a client website, so I used a series of personal blogs within my network. I used four variables to test the results of NoIndexing taxonomies. Before the experiment, all four blogs had all taxonomies enabled for indexation and none of them had optimized pagination.

Traffic growth had been slow and sluggish on all four sites. On variable one, I NoIndexed all taxonomies and left the pagination unoptimized. Variable two had optimized pagination, but all taxonomies were allowed to be indexed. Variables three and four had both optimized pagination and all taxonomies set to NoIndex.

Variable One: NoIndexed Taxonomies and Unoptimized Pagination

For the first variable, I used www.midwesternadventures.com, which belongs to a good friend of mine. Although the website has undergone a redesign recently, many elements, including the pagination of the website, were written in jQuery with the old design during this experiment. This made way for a perfect un-optimized pagination, NoIndex variable since it would be difficult to crawl the pagination anyway. Google’s index of the site continued to remain steady despite shallow crawl depth.

google index steady No Indexing WordPress Taxonomies: Do or Don’t

Google’s index of midwesternadventures.com

Within two weeks of NoIndexing the taxonomies, organic traffic plummeted by nearly 20 percent, while Google’s index of the website remained stable.

Since the pagination was written in jQuery and GoogleBot has a hard time crawling jQuery, the crawl depth of the website was very shallow and all pages not linked from the homepage began dropping dramatically in organic traffic, since authority was not being passed along properly between each post.

After two weeks of starting the experiment, I decided to redesign the site and optimize the pagination since traffic dropped so greatly. Traffic skyrocketed after the redesign and optimization.

Variable Two: Indexed Taxonomies and Optimized Pagination

google index increase No Indexing WordPress Taxonomies: Do or Don’tGoogle’s index of thetravelpixie.com

The second variable was www.thetravelpixie.com, which belongs to the same friend. The only thing I changed on this site was optimized pagination. Since the site still had massive amounts of duplicate content from taxonomies, organic traffic growth continued to be sluggish.

Some days it would grow by 3 percent, while other days it would lose 3 percent. Having optimized pagination alone didn’t have much effect on traffic. Google’s index continued to grow as a result of the optimized pagination, but traffic remained sluggish.

Variables Three and Four: NoIndexed Taxonomies and Optimized Pagination

The third and fourth variables yielded the most interesting results of all variables. Variable three was www.bizarreworldfoods.com, which is a site me and the same friend run together. Since it is still a relatively new site, I felt it necessary to run a second variable in this category.

The fourth variable was my own Chicago food blog, www.hgjones.org, which is a well-established site. Within a few days of optimizing the pagination and NoIndexing all taxonomies, Google’s index of the sites plummeted.

google index plummet No Indexing WordPress Taxonomies: Do or Don’t

Google’s index of hgjones.org

Although the index of variables three and four plummeted, organic traffic skyrocketed. Traffic on variable three increased by 30 percent within two weeks, while variable four increased by 20 percent.

organic traffic growth 1024x149 No Indexing WordPress Taxonomies: Do or Don’t

Organic traffic of hgjones.org

Conclusion

As a result of this experiment, I discovered that NoIndexing taxonomies and optimizing pagination will give a massive boost in organic traffic within a few weeks of implementation. Google and Bing recently placed duplicate content in their crosshairs with recent algorithm updates. It is important for webmasters to avoid duplicate content, and to ensure taxonomies are not causing a lack of growth.

Optimized pagination takes the place of taxonomies in crawl depth, so including the rel=prev and rel=next attributes are vital to optimizing taxonomies.

WordPress Plugins Used

I used two WordPress plugins for this experiment, which I highly recommend anyone running a WP site to use. The first plugin I used is called WP-PageNavi, which adds optimized pagination links using the rel=prev and rel=next attributes. This plugin may require some code implementation, but the plugin developer provides all code needed.

The other plugin I used is called WordPress SEO by Yoast, which is the best SEO plugin available in my opinion. Many webmasters will also use All-In-One SEO, which I personally believe is inferior to WordPress SEO. Yoast’s plugin has import controls for importing All-In-One’s settings into WordPress SEO. Instructions for migrating to WordPress SEO can found here.

To NoIndex taxonomies with the WordPress SEO plugin, go to your WP dashboard and click on SEO in the sidebar. Select Titles &Metas. On the General tab, check the box to NoIndex subpages of archives. On the Taxonomies tab, check the box to noindex, follow categories, tags, and format. On the Other tab, check the box to noindex, follow author archives and date archives. Unless you have multiple authors, I would also recommend disabling author archives completely.

 No Indexing WordPress Taxonomies: Do or Don’t

Harrison Jones

VP of Digital Marketing at MWI
Harrison Jones is the VP of Digital Marketing at MWI - a global search, content marketing, PR and conversion optimization agency. He specializes in SEO, paid search, analytics, usability analysis, information architecture, conversion optimization, and content marketing.
 No Indexing WordPress Taxonomies: Do or Don’t

You Might Also Like

Comments are closed.

27 thoughts on “No Indexing WordPress Taxonomies: Do or Don’t

  1. Excellent research Harrison, I’ve always noindexed taxonomies myself, although been aware that many still think it’s more damaging doing so. Whilst I don’t think this’ll be the final word on the matter it’s good to see others have had similar experiences to myself.

    1. Thanks for the compliment. NoIndexing taxonomies can be risky if the website architecture is not set up for such a move. As long as the website has optimized pagination and breadcrumbs, noindexing the taxonomies will help with search performance because with the pagination and breadcrumbs, there is no use for taxonomies for crawl depth.

      1. Harrison,

        I have a quick question here. I fail to understand what “optimized” pagination really mean? The homepage does have a pagination powered by WP Pagenavi (with rel next and prev of course). Is that all one needs to do?

        Also, when we use Yoast’s SEO plugin, the second page (domain.com/page/2) and the rest of the them are, by default, noindex and followed. Is that a good thing?

        Thanks for writing this up — immensely helpful! :)

  2. Great article. When you say pagination you mean what? How the page structure is setup? Do tags make dupe content as well as categories? I was always taught to do as many tags as you want but only one category. Was that wrong?

  3. I’m slightly confused with the findings to be honest, what am I missing here? If Google’s indexing dropped in variables three and four due to NoIndexed Taxonomies but, traffic increased overall surely that means that the site is more focused and less diluted with the tags or extra keyword noise therefor, Google gives it more weight and focus on its content thus you get an increase in traffic?

    1. Basically, when you allow taxonomies to be indexed, you are creating more pages for Google to index. However, those pages are full of duplicate content which devalues your actual post pages and website overall. Even though noindexing the taxonomies decreases the amount of pages in Google’s index, the pagerank of your posts increases because it is seen as original content rather than a copy. It also creates a more focused SERP listing. I’m sure you’d rather have users land on your actual post page rather than an author archive or category page. Duplicate content has been in the crosshairs of Google lately, and penguin was a huge move towards devaluing sites and pages with dupe content.

  4. Interesting topic. It’s hard to make general recommendations on this, though. The usefulness of taxonomies for search purposes is likely to depend on the number of topics and the amount of content for each topic. If you have lots of different topics, it could help to have taxonomy pages indexed, but only if each topic has considerable content. The key factor is that, if your taxonomy pages have few posts, they will look too similar to the original pages; while taxonomy pages with multiple posts will appear more unique.
    Hence I think the decision should be made on a case-by-case basis. The same applies to the choice of tags or categories (keeping both can be more prone to dupe content).

    1. Great point you made there. In the event you would actually want your taxonomies indexed, the best course of action would be to only have summaries of your posts within the taxonomies rather than the full version of the post. This will cut down on the dupe content factor, but there will still be a mild conflict. However, if you do allow taxonomies to be indexed, I would strongly recommend writing unique title tags and descriptions for them so they do perform better in SERPS

  5. Now this is the sort of content I want to read as an SEO. Great article and I’ve always wondered this question but never felt I had the time or resource to perform such an experiment. Bookmarked. Really useful stuff.

    1. Thanks Dave. I have had many inquiries over the last few months about this technical SEO issue, so I decided to run an experiment on it. Although Yoast states in his WP SEO guide to noindex all taxonomies, I wanted to analyze his statement and give real data behind his theory. His theory proved to be correct as long as other structural measures are taken like breadcrumbs and optimized pagination. I have more experiments in the works right now, which I plan to publish in the future.

  6. Hi Harrison
    Thanks for sharing your experiment.
    I always no index taxonomies myself too. One question though:
    I do not undersand why using the WP-PageNavi plugin as WordPress seo by Yoast manage the rel next rel prev pagination
    Is there other uses for the WP-PageNavi plugin?
    cheers

    1. The reason I use WP-PageNavi is because Yoast highly recommends using it on his WordPress SEO guide. If he is recommending another plugin over his own, I felt it would be better to use it instead.

  7. Good stuff this, always good to have tidy content regarding search engines.

    One Q if i may, i’m getting domainname.com/page/feed showing up as crawl errors in webmaster tools…have turned off indexing as per above and other Yoast recommendations though. Any ideas where /feed is coming from? Does it matter? thx

    1. That is a rather common bug with WordPress. I’m not much of a WordPress developer, so I couldn’t tell you how to fix it. I would recommend setting up a 301 redirect for the feed page back to the page itself. I would also go into Google Webmaster Tools and use the URL Removal tool for that feed URL so it gets removed from Google’s index. It does matter to take care of that issue because Google is a stickler on crawl errors, so it’s best to remove any errors you come across.

  8. Thanks for a great article! I was weeping hot tears on my keyboard trying to figure that one out today! I have a Woocommerce based shop and am getting crawl errors from all the Variations on deleted items and deleted tags. Will noindexing all the taxonomies stop that? I don’t want to have to build 700 301 redirects…

    Thank you again for taking the time to write and share this.

    Kathy

  9. Thanks for this. I’ve been looking everywhere for an explanation of why Google is indexing many more pages than I have actually published and now I have the answer. One quick question. If I disable the Author Archives will this affect my picture appearing in SERPs?

  10. Hi Harrison, thank you very much for this great help.

    I want to know, I use Genesis, should use the pagination of Genesis or is it better to use the WP-PageNavi

  11. Awesome article and the results are impressive to say the least.
    I just started a new website that relies heavily on custom taxonomies for categorizing content. I do not use categories and tags, so I have them set to noindex, follow. Would you say that noindexing the rest of my custom taxonomies will have the same positive impact on the organic traffic? I have proper, optimized pagination by using the WP-Pagenavi plugin. Thank you!

  12. Great post Harrison,
    I see this was posted a year ago and assume you still recommend no-indexing all that you mentioned in the post?
    Thanks

  13. NIcely done – good (and rare) to see some real world examples. I’m also curious as to whether this is the best option with the most recent algorithm changes. Can you possible give us your thoughts on this?