While Google is experimenting on crawling hidden web pages through HTML forms indexing, Yahoo on the other hand has updated its search crawler with Slurp 3.0. Although the implementation of Slurp 3.0 would not really pose a big implication on webmaster’s part, just the same here are the changes that Slurp 3.0 will bring in the way it will crawl websites.
First, Slurp 3.0 will start crawling from smaller set of IP addresses, although still within crawl.yahoo.net.domain. Reverse DNS checks will still continue working. For webmasters who use IP-based recognition for identifying Yahoo crawlers, Yahoo advises to move to reverse DNS-based identification of Yahoo! Slurp to avoid getting dropped by the Yahoo Slurp 3.0 crawlers.
Second, Yahoo! Slurp 3.0 will now publish a new user-agent – “Yahoo!Slurp 3.0”. Although existing robots.txt directives for “Slurp” or “Yahoo! Slurp” will continue working, directives for “Slurp 2.0” would not work anymore. So, Yahoo suggests that webmasters use the shorter version of the User-agent which is simply – Slurp.
I wonder if this is just a coincidence or if it has to do with their new bot; today is the first time that the Yahoo! cache of my site has been more than 1 week fresher than Google’s cache of my site.
What is equally impressive is the less than 24 hour period it now takes yahoo to index new social sites (including the sites that are linked TO from then – even if they are using the NOFollow tag)
During SES 2008 NYC – keynote – The Yahoo commentator did announce that Yahoo would be incorporating more of the social web in SERPs.
You can see the influence now – where social sites are high on the search results
it takes yahoo too long to crwal sites sometimes any ideas on how to bypass that?
Social sites are now taking over
i suggest twitter.