When IE9 hit the market, one of the more interesting controversies in web browser releases erupted: Microsoft was claiming it was faster at performance benchmarks, and competitors said this simply wasn’t so. This wasn’t a case of Google accusing Microsoft of cheating (although they’ve accused them as much with search engine results). Rather, this was a set of competitors and analysts reminding users that Microsoft was using its own benchmarks. Meanwhile, results on actually performance benchmarks is often inconclusive.
To help remedy this, the Google Chromium team put together modified versions of two major benchmark suites: SunSpider, developed by Apple, and Kraken, developed by Mozilla. Both of these JavaScript benchmark tests evaluate important areas forJS web standards, but both – according to the Chromium team – are flawed in at least some way. First, SunSpider is old enough that browsers have basically mastered its tests; tests are finished in a few milliseconds, leaving far too much variability time based on system speed, internet connection, and other elements that aren’t meant to be a part of the test. And second, Mozilla’s Kraken has failed to resolve some bugs that throw off test results.
Google addressed the SunSpider issue by having it run the same test 50 times, back to back. The hope is that this alteration will allow for more accurate results since the broader scope should diminish the impact of variables. Kraken, meanwhile, was re-constructed by the Chromium team from the Mozilla Source Code Repository, hypothetically eliminating the bugs in question. When tested under these circumstances, Google finishes tests up to four times faster than competitors, and 30% faster overall on the SunSpider suite. Users can download Google’s versions of SunSpider and Kraken, but in the end Google is still creating the tests that show their browser as superior.
[via the Chromium Blog]
