Before moving towards X-Robots-Tag, let’s have a small discussion on Robots. As webmasters, we already know about this term. It allows you to hide pages, folders, sub-domains or any content from a search engine spider.
There are two ways you can use robots.
Robots.txt: It is a text file where you can specify your content to be hidden from spiders. You can find it on root domain or sub-domain.
You can also add sitemap for a quick index purpose by search engines.
Robots Meta Tag: The robots meta tag gives you a page-specific approach to controlling how an individual page should be indexed and served to users in search results. You will have to put it <head> section of a given page.
What is X-Robots-Tag?
The X-Robots-Tag can be used as an element of the HTTP header response for a given URL of the web page.
How can I instruct crawlers not to index a page?
Benefits of Using it:
- You can use it where the usage of robots meta tags is not possible, for example: non-html files like Image, Video, and Flash can be blocked.
- You can add the X-Robots-Tag to a site’s HTTP responses using .htaccess and httpd.conf files.
- It is Global, which means an X-Robots-Tag with HTTP responses allows you to specify crawling directives that are applied globally across a site.
- You can use regular expressions for high quality flexibility for complex url or content.
- Like Robots Meta Tag, you can use other attributes for your page like nosnippet, noodp, notranslate etc.
If you do not want to use robots.txt or robots meta tag, or need to block non-HTML content, then use X-Robots-Tag. Check the announcement from Google about X-Robots-Tag here.