Comments on: Yahoo Blocking Bots from Spidering Delicious Bookmarks

By: Farhad Divecha

Farhad Divecha — Thu, 21 Feb 2008 20:35:06 +0000

Thank you Michael.

Ugh! I’m tired of misinformation being spread by self proclaimed “SEO Analysts”. This is just plain false.

Michael is right. The only pages being blocked are admin type pages and spoofers

By: Pratheep

Pratheep — Tue, 19 Feb 2008 17:10:34 +0000

Yes, lets wait for any inputs from delicious.

Pratheep

By: Michael VanDeMar

Michael VanDeMar — Tue, 19 Feb 2008 17:04:58 +0000

Although, to be fair, he might actually be right. Just because the logic has a hole in it doesn’t mean that they didn’t actually block em. We just have to wait and see.

By: Michael VanDeMar

Michael VanDeMar — Tue, 19 Feb 2008 16:50:27 +0000

Pratheep, everyone is citing Colin’s article though. If the initial source is incorrect in their assumptions (which is what I am saying I think to be the case), then it doesn’t matter how many people repeat them… they would all be wrong.

I’m neither the first nor the only person to point out that they are probably checking by IP, and only blocking that user agent from invalid networks.

By: Pratheep

Pratheep — Tue, 19 Feb 2008 16:46:17 +0000

But Michael, in seroundtable.com there is a thread running about the same topic, they are discussing somthing about the search engine user-agent getting 404 error. Please take a look at the following post,

http://www.seroundtable.com/archives/016276.html

Pratheep

By: Michael VanDeMar

Michael VanDeMar — Tue, 19 Feb 2008 05:50:56 +0000

Pratheep, /post is just where you submit new urls. It’s a blank form, no reason to index it.

By: Pratheep

Pratheep — Tue, 19 Feb 2008 05:48:41 +0000

Sorry about my previous comment, I have pasted a wrong URL,

http://209.85.175.104/search?sourceid=navclient-ff&ie=UTF-8&q=cache%3Ahttp%3A%2F%2Fdel.icio.us%2Frobots.txt

Please check now.

Pratheep

PS: Administrator of this blog: please remove that URL from my previous comment.

By: Pratheep

Pratheep — Tue, 19 Feb 2008 05:46:49 +0000

Yes, Michael you are correct!

Here the cache URL for their robots.txt file,

http://matt-cutts.blogspot.com/2008/02/google-update-and-other-interesting.html

But why they have blocked “post” directory?

Pratheep

By: Michael VanDeMar

Michael VanDeMar — Mon, 18 Feb 2008 16:02:29 +0000

Loren, neither the bookmarks nor the profiles linking to the sites are blocked. The bookmarks are all under the /url/ directory, and the profiles are all under root. The only things blocked are the unimportant things that del.icio.us always blocked.

Look at the cache of robots.txt, and you can see that it hasn’t changed since Dec 24th.

He misinterpreted what he saw.