robots.txt
1: A text file, accessible at the root of a website, that tells search engine crawlers which areas of a website should be ignored.
FAQ
Why is having a robots.txt important?
A robots.txt file is important so crawlers or bots that visit your site know which pages can be indexed. It blocks bots from crawling certain pages you don’t want to be listed.
What is a user agent?
A User agent is the web crawler that you are giving instructions to.