Advertisement

Top Definition

robots.txt

1: A text file, accessible at the root of a website, that tells search engine crawlers which areas of a website should be ignored.

Advertisement
Continue Reading Below

FAQ

Why is having a robots.txt important?

A robots.txt file is important so crawlers or bots that visit your site know which pages  can be indexed. It blocks bots from crawling certain pages you don’t want to be listed.

What is a user agent?

A User agent is the web crawler that you are giving instructions to.

Advertisement
Continue Reading Below

Related Terms

Advertisement
Continue Reading Below