Yahoo just got a step closer to dropping the Google search results from its search function and replacing them with Yahoo’s own Inktomi search engine- which will be a bit of a blow to Google, and a sign of potential dominance by Yahoo.
Yahoo has just unleashed a new site indexing robot to crawl the web with – Yahoo! Slurp.
Yahoo’s new robot keeps a similar name to the Inktomi Slurp crawler and some features listed on Yahoo include:
* Yahoo! Slurp has the ability to crawl dynamic links or dynamically generated documents.
* The Yahoo! Slurp crawler collects documents from the web to build a searchable index for search services using the Yahoo! search engine (this helps verify a soon addition of Inktomi to the Yahoo search results). These documents are crawled since other documents on the web contain links to these documents.
* Yahoo! Slurp crawls from your site in the Yahoo! search engines immediately. The documents will be indexed and included into the search database in the near future.
* Yahoo! Slurp will offer cache indexing (similar to Google) and obeys the noarchive meta-tag. If you place: META NAME=”robots” CONTENT=”noarchive” in the head of your web document, Yahoo! Slurp will retrieve the document, but it will not cache or archive the document for use in the PageCache system.
* Yahoo! Slurp also obeys the noindex meta-tag. If you place: META NAME=”robots” CONTENT=”noindex” in the head of your website, Yahoo! Slurp will retrieve the document, but it will not index the document or place it in the search engine’s database.
Last year, Yahoo announced that they plan to make the change over to Inktomi results in the first quarter of 2004, which gives them about 40 days to meet that goal. In addition, its nice to see that the Slurp robot and search functions are all branded Yahoo.
It gives it more of a unified feel to have all of its main functions together, indivisble, under the united brand umbrella of Yahooooo!
Seen Yahoo Slurp in your server logs? What do you think? Leave a comment.