Start Now!
Advertisement

Yahoo Search’s Slurp Crawler Updates Moves to New Address

  • 272
    READS
Yahoo Search’s Slurp Crawler Updates Moves to New Address

Yahoo has updated and moved its Yahoo Slurp crawler to a new domain at crawl.yahoo.net, but is keeping the same IP address for those publishers which use IP filtering, so there should be no need for publishers to edit their robots.txt files.

Yahoo Search adds that publishers should setup reverse DNS authentication to ensure that no false Yahoo Slurps visit their sites:

With this transition complete, we also encourage you to setup reverse DNS-based authentication of our crawler to ensure that no rogue bots masquerading as ‘Slurp’ visit your site. Here is how it works:

    1. For each page view request, check the user-agent and IP address. All requests from Yahoo! Search utilize a user-agent starting with ‘Yahoo! Slurp.’
    2. For each request from ‘Yahoo! Slurp’ user-agent, you can start with the IP address (i.e. 74.6.67.218) and use reverse DNS lookup to find out the registered name of the machine.
    3. Once you have the host name (in this case, lj612134.crawl.yahoo.net), you can then check if it really is coming from Yahoo! Search. The name of all Yahoo! Search crawlers will end with ‘crawl.yahoo.net,’ so if the name doesn’t end with this, you know it’s not really our crawler.
    4. Finally, you need to verify the name is accurate. In order to do this, you can use Forward DNS to see the IP address associated with the host name. This should match the IP address you used in Step 2. If it doesn’t, it means the name was fake.
CategorySEO
ADVERTISEMENT

Subscribe to SEJ

Get our daily newsletter from SEJ's Founder Loren Baker about the latest news in the industry!

Ebook

Loren Baker

Loren Baker is the Founder of SEJ, an Advisor at Alpha Brand Media and runs Foundation Digital, a digital marketing ... [Read full bio]

ADVERTISEMENT
Advertisement
Read the Next Article
Read the Next