A lot has been said recently about different ways to keep your content hidden from search engines. Still there is much confusion regarding which method serves which aim. So I decided to make a quick easy-to-understand guideline of what should be done depending on what you need to achieve.
So mainly you might want a search engine to treat your content in 4 possible ways:
- crawl, index and rank your content;
- do not crawl but (possibly) return in search results (i.e. rank);
- do not index and consequently do not rank;
- do not follow the given link only from this very page.
While with the first option you do not generally need any directives; with 3 other aims the right techniques would be:
- do not crawl => robots.txt directive (NOTE: the page might still be indexed and ranked when discovered via external links pointing to it);
- do not index => ‘noindex’ directive attached to the page (NOTE: if you use both robots.txt disallow directive and noindex meta tag, the page may still be indexed: a search engine won’t crawl the page and hence will be unaware of the noindex directive; so in this case ‘more’ doesn’t mean ‘better’);
- do not follow the link => rel=”nofollow” attribute attached to a link (NOTE: if there are other ‘dofollow’ external/internal links to the page, it will be crawled, indexed and ranked).
Sebastian had also a great piece published at SEOMoz blog discussing crawler and indexer directives and how they are treated by different search engines.
Comments
7 responses so far ↓
doug m on Apr 22, 2008 at 11:39 am
this is a great article. now how would I stop links from showing up that have no content, IE: certain php pages that don’t related to my posts
Louis Liem on Apr 22, 2008 at 12:31 pm
so, correct me if i’m wrong..
noindex means not displaying the page on SERP and nofollow means not following the link and in google’s case, not giving any PR link juice?
Name changed to protect the Sphinner on Apr 22, 2008 at 2:33 pm
I’ve long wondered whether to add “noindex,follow” to the front page of my blog to avoid a duplicate content penalty.
The problem is that I have links from social networking profiles pointing there and so its gaining (a little) rank.
I read that search engines understand blogs and don’t impose this penalty - but do the SEOs out there have a definitive answer?
chris on Apr 22, 2008 at 9:20 pm
good job!
kokotaro on Apr 22, 2008 at 9:30 pm
thanks, now i can understand completely the crawl, index and rank.
but i always feel confused that what’s the differences between “external nofollow” and “nofollow”.
i really want to know the answer. thanks again.
Matt on Apr 25, 2008 at 7:20 pm
Nofollow does not prevent engines from following the link. You are mistaken. The link is still followed. All it does is prevent passing of link juice.
Ann Smarty on Apr 26, 2008 at 4:18 am
@Matt - no, actually a link with Nofollow attribute won’t be followed. You can run a simple test by creating a new page at your site and nofollow all links to it. The page won’t be crawled and indexed until you create a “dofollow’ link to it.
Here is also quite a straightforward citation by Matt Cutts if you don’t trust me:
“NoFollow as an individual link attribute means don’t follow this particular link…”
http://www.stonetemple.com/articles/interview-matt-cutts.shtml
Leave a Comment