MSN Search Beats Google, Yahoo & Ask.com in Search Engine Relevancy Study
According to a recent study perfomed by Intralink, MSN Search ranked as being more relevant than Google, Yahoo, Ask.com, AOL, Gigablast & Wisenut.
Before everyone’s veins start popping out of their heads and the comments on this post flame up with MSN Search Spam examples and tales of SEO’s manipulating MSN; let’s take into account the variables used for judging “relevancy” by this Intralink company:
Relevancy, freshness of content, failure rate, difficult search results, and non-organic or extra features.
In terms of freshness, Google and MSN showed the most up to date indexes. And although the V7N contest and launch of Umdum have showed Yahoo, Ask.com, and MSN to index sites faster at the times of site launches – Yahoo, Ask.com and Gigablast ranked worst in overall Freshness.
On SEOResourceCenter.com, Intralink has published a breakdown of their study and relevancy scores:
Relevancy (250 total searches)
The first and most important criteria was could a result be found on the first page. Additional points were given for additional exact matches on the first page. But because relevancy is somewhat in the eye of the beholder an exact match of 1 on the first page rated much higher than a 3 on the second or third page. Additionally 4 exact matches doesn’t way much more than 3. Then we looked at the total number of exact matches for the term. Because so many searches are unique we weighed the difficult searches higher. In other words, who performs best at the difficult searches? is essentially the same as asking, “which search engine is most likely to find the best results for my unique search?”,
Freshness of content (30 searches)
This is a two part test. The first is for a current event (American idol results or specific lotto results in the last week could the search engine produce the information desired. All searches engines tested with the exception of Gigablast and Wisenut received perfect scores. Wisenut was close missing perfect by just 10%. Gigablast was penalized because they only matched about 20% of the time. The second factor was how recent was the content displayed. A couple of example searches were; “condoleezza rice article” and “iraq war”.
Google did very well with freshness on news related searches. But Google and Yahoo! both had problems finding new businesses. Both of these search engines apparently employ a filter on new web sites. In Google this has been referred to as the sandbox effect. On the surface it makes sense. The rumor is that Google puts a new web site on the back burner for a few months to determine if the site is going to stay around, or if it’s something someone has thrown up to try to test, or make money quickly. The problem comes in to play when a new restaurant or auto dealer, or any other useful site is launched. If you only use Google or Yahoo! for your search it could take months to find it. For more information on this effect WebMasterWorld has forum where the topic is discussed. Here is link to one section of that forum.We are currently working on the next search engine relevance report which will be released in July. We are adding a controlled search study to factor in newly released web sites.
Failure rate (250 total searches)
How many times was the search engine not able to produce the information in the first 3 pages of a difficult search. A difficult search was defined as at least 3 of the search engines not finding the result on the first page. This is particularly frustrating to users and received a heavier weighting than extras (like maps, or local searches).
Difficult searches results (40 searches)
How well did the search engine do when first page results were hard to come by. A couple of example searches are “car dealer fargo north dakota” and “appliance repair des moines”.
Non-organic and extra features.
This is the most subjective and weighted the lightest overall. We peformed spot checks on search engines with maps, who could find a specific address? was an example. We also looked at the local results searches. This are directories that make it easy to find a phone number or location. Most of the local search features worked fairly well but Googles was an exceptional feature. In search after search the local feature was 100% accurate. If we did a local lookup on a Cincinnati car dealer, we got car dealers in Cincinnati. In other search engines we were shown results for car repair, tire and brake shops and other car related listings that were not car dealers.
Intralink also lays down their search samples, an overview of the report by CEO Eric Gurr, and why they performed the research.
This study may have not gotten as much mention in SEO circles in the past, but Intralink’s timing is amazing and incredibly relevant to the search engine world this week, esecially with MSN Live Windows now powering Amazon.com, Ask.com’s Steve Berkowitz moving over to Microsoft Search, and the Google Firefox / MSN Search Windows Vista ‘controversy’ sparking up.
Do you agree with Intralink? Is MSN Search now a better all around search engine than Google, Ask.com or Yahoo?
