Yahoo has updated and moved its Yahoo Slurp crawler to a new domain at crawl.yahoo.net, but is keeping the same IP address for those publishers which use IP filtering, so there should be no need for publishers to edit their robots.txt files.
Yahoo Search adds that publishers should setup reverse DNS authentication to ensure that no false Yahoo Slurps visit their sites:
With this transition complete, we also encourage you to setup reverse DNS-based authentication of our crawler to ensure that no rogue bots masquerading as ‘Slurp’ visit your site. Here is how it works:
1. For each page view request, check the user-agent and IP address. All requests from Yahoo! Search utilize a user-agent starting with ‘Yahoo! Slurp.’
2. For each request from ‘Yahoo! Slurp’ user-agent, you can start with the IP address (i.e. 74.6.67.218) and use reverse DNS lookup to find out the registered name of the machine.
3. Once you have the host name (in this case, lj612134.crawl.yahoo.net), you can then check if it really is coming from Yahoo! Search. The name of all Yahoo! Search crawlers will end with ‘crawl.yahoo.net,’ so if the name doesn’t end with this, you know it’s not really our crawler.
4. Finally, you need to verify the name is accurate. In order to do this, you can use Forward DNS to see the IP address associated with the host name. This should match the IP address you used in Step 2. If it doesn’t, it means the name was fake.
Category
SEO
