Check Your Link History Now

X-Robots-Tag: A Simple Alternate For Robots .txt and Meta Tag

  • 34
  • 14K
X-Robots-Tag: A Simple Alternate For Robots .txt and Meta Tag

Before moving towards X-Robots-Tag, let’s have a small discussion on Robots. As webmasters, we already know about this term. It allows you to hide pages, folders, sub-domains or any content from a search engine spider.

There are two ways you can use robots.

Robots.txt: It is a text file where you can specify your content to be hidden from spiders. You can find it on root domain or sub-domain.


User-agent: *


You can also add sitemap for a quick index purpose by search engines.

Robots Meta Tag: The robots meta tag gives you a page-specific approach to controlling how an individual page should be indexed and served to users in search results. You will have to put it <head> section of a given page.


Now lets talk about the  X-Robots-Tag. First,  let me be clear that if you are using above robots.txt and robots meta tag for blocking your content, then there is no need to use this one.

What is X-Robots-Tag?

The X-Robots-Tag can be used as an element of the HTTP header response for a given URL of the web page.

How can I instruct crawlers not to index a page?

How can I show my page on search results if I am following some conditions?


Benefits of Using it:

  • You can use it where the usage of robots meta tags is not possible, for example: non-html files like Image, Video, and Flash can be blocked.
  • You can add the X-Robots-Tag to a site’s HTTP responses using .htaccess and httpd.conf files.
  • It is Global, which means an X-Robots-Tag with HTTP responses allows you to specify crawling directives that are applied globally across a site.
  • You can use regular expressions for high quality flexibility for complex url or content.
  • Like Robots Meta Tag, you can use other attributes for your page like nosnippet, noodp, notranslate etc.

If you do not want to use robots.txt or robots meta tag, or need to block non-HTML content, then use X-Robots-Tag. Check the announcement from Google about X-Robots-Tag here.


Featured Image:  Rawpixel via Depositphotos


Subscribe to SEJ

Get our daily newsletter from SEJ's Founder Loren Baker about the latest news in the industry!


Chandan Das

Owner and Co-founder at Clicks4info

Chandan Das is the Owner and Cofounder of Clicks4info, Who has interest in Blogging, SEO, SMO and also spend time ... [Read full bio]

Read the Next Article
Read the Next