Comments on: Robots.txt : 4 Things You Should Know

By: Alex

Alex — Tue, 09 Mar 2010 11:22:48 +0000

Very usefull info! But I’m a bit confused. Imagine having the Following robots.txt:

User-agent: gsa-crawler
Disallow: /careers/user/

# Disallow the rest
User-agent: *
Disallow: /

Will it be interpreted the same way as:

# Disallow the rest
User-agent: *
Disallow: /

User-agent: gsa-crawler
Disallow: /careers/user/

…and will the folder ‘/careers/’ be crawled in both cases?

By: sam

sam — Mon, 08 Dec 2008 18:17:57 +0000

Nice post Ann!
Thanks.

By: Shanghai Web Hosting

Shanghai Web Hosting — Wed, 16 Jul 2008 02:51:59 +0000

Nice tips, thanks for sharing. Robots.txt are one of those small things that are usually ignored by webmasters.

By: John H Gohde

John H Gohde — Tue, 15 Jul 2008 12:29:15 +0000

What I find to be the most overlooked tidbit on robots text files is that if you want to remove a webpage from from Google, excluding it in your robots.txt can actually be counter productive.

By: J Solutions 網頁設計

J Solutions 網頁設計 — Tue, 15 Jul 2008 11:46:46 +0000

robots.txt can be very harmful if set wrongly. So I will recommend to use Google Webmaster Tool to test it before uploading to your website.

By: zemin temizligi

zemin temizligi — Tue, 15 Jul 2008 10:13:53 +0000

thank you….

By: Software Testing

Software Testing — Tue, 15 Jul 2008 07:50:48 +0000

Thanks Ann!

By: istioselida

istioselida — Tue, 15 Jul 2008 07:05:51 +0000

very useful post, i will use your tips to add the file in my sites!

By: Marcel

Marcel — Tue, 15 Jul 2008 00:06:04 +0000

Good overview. I learned a couple of things – thanks.

By: Garrett Pierson

Garrett Pierson — Mon, 14 Jul 2008 19:15:27 +0000

Wonderful tips Ann! Especially tip # 4 “you’d better have Robots.txt file even if you do not want to include any (specific) directions”. I toally agree with this and have always taught my clients and students how important this is. Thanks again for the great content and tips!