Alexa Bias Exposed by Top Google Engineers

SMS Text

Alexa, the web site popularity and ranking measurement metric, has been widely accepted by the web publisher and online advertising world as a way to gauge the popularity of a site without viewing its log files.

In fact, Alexa rankings have become an important part in the formula of many online advertising firms, which feel that sites with a higher Alexa ranking should be rewarded more in terms of CPM or link advertising pricing.

Alexa is Not Perfect

There has been a mutual understanding in the web community however, that Alexa rankings are not accurate and are biased towards sites about web marketing, search marketing, and web site publishing.

Why? Because a majority of Alexa toolbar users download the tool to check the rankings of their own site, and with hope that the more they visit their site with a browser equipped with the Alexa toolbar, the higher their Alexa ranking will become.

Alexa is Biased

Alexa has addressed some of its biases:

…the Alexa user base is only a sample of the Internet population, and sites with relatively low traffic will not be accurately ranked by Alexa due to the statistical limitations of the sample. Alexa’s data come from a large sample of several million Alexa Toolbar users; however, this is not large enough to accurately determine the rankings of sites with fewer than roughly 1,000 total monthly visitors. Generally, Traffic Rankings of 100,000+ should be regarded as not reliable because the amount of data we receive is not statistically significant. Conversely, the more traffic a site receives (the closer it gets to the number 1 position), the more reliable its Traffic Ranking becomes.

I’ve discussed in the past how easy it is to game and manipulate a site’s Alexa ranking and reasons why Amazon should look into acquiring other toolbar companies which would also report on wider site usage and deliver more fair and balanced Alexa results.

Still, there is not a tool better than Alexa in terms of ranking web site usage and popularity on the market, and until there is, advertising firms and web publishers will overlook the Alexa bias.

Alexa Skewed Towards Webmaster & Search Marketing Sites

Google Engineers Peter Norvig and Matt Cutts recently looked at the Alexa numbers for their sites and put together an Alexa webmaster skew.

Peter, who is the Director of Research at Google, compared his site’s Alexa score with those of Matt Cutts, Paul Graham, Jeremy Zawodny and Greg Linden to find a definite reporting difference for sites that are search or web marketing related and those which are not.

Norvig writes at Alexa Toolbar and the Problem of Experiment Design:

… I get about twice the pageviews of, but his Alexa pageview ranking is about 25 times more than mine (I got this by looking at the 1 year, most highly smoothed graph, and then squinting to guess at the mean).

What that means is that people with the Alexa toolbar installed are 25 times more likely to view a page on Matt’s site versus mine, but overall, all users view twice as many pages on my site.

That’s a 50 to 1 difference introduced by the selection bias of Alexa. Presumably this is because Matt’s site is really appealing to a core group of SEO enthusiasts, many of whom also like the Alexa toolbar.

Norvig adds that Alexa does not address this weakness, which is its Selection Bias:

The data would be good if it truly represented a random sample of internet users, but in fact it only represents those who have installed the Alexa toolbar, and that sample is not random.

The samplees must be sophisticated enough to know how to install the toolbar, and they must have some reason to want it. It turns out that the toolbar tells you things about web sites, so it is useful to people in the SEO industry, so it overrperesents those people.

Alexa Alternatives?

Just because Peter Norvig and Matt Cutts look into Alexa’s bias and site stat tracking does not mean that Google is going to launch the next Alexa.

Sure, they could. But isn’t site popularity part of the PageRank equation anyway?

In saying that Google could launch the next Alexa, I’m saying that if tomorrow Google all of the sudden adds a site traffic metric similar to and better than Alexa, they’d simply blow Alexa out of the water. But I do not believe Google has the intentions nor wants to share such data which is tracked by their toolbar, AdSense ads, and post search behavioral tracking. I’m sure Google would rather keep those metrics internal, which will better the way their search algorithm works and their targeting of web, video and audio advertising.

Yahoo, with the snap of their fingers, could also do the same.

Instead, one of the major Alexa competitors out there is Compete.

Sure, Compete’s numbers are a bit off as well (SEJ received triple the amount of users that Compete reports in January), but they are doing a fine job at reinventing the mousetrap with user metrics, traffic rank, pages per visit and average stay.

For more information on how Compete calculates their ranking and traffic information please see their FAQ page.

Besides Alexa, Google PageRank and Compete, are there any other tools you use to gauge the traffic and value of a web site or your competition? Please feel free to share them below.

Loren Baker
Loren Baker is the Founder of SEJ, an Advisor at Alpha Brand Media and runs Foundation Digital, a digital marketing strategy & development agency.
Loren Baker
Subscribe to SEJ!
Get our weekly newsletter from SEJ's Founder Loren Baker about the latest news in the industry!
  • Patrick C

    I’d really like to see actually compete with Alexa in terms of site owner and advertiser adoption of their metrics numbers.

    I also liked SEOmoz’s PageStrength, but it is dependent upon Alexa, Google & Yahoo for its own numbers, so it’s not standing on its own two feet.

  • Chris S

    Alexa ranking is really poor, and it always kills me whenever I see someone use it as a metric for judging a site. For example, Text-Link-Ads uses it as one of its criteria for allowing new publisher sites to join…

    Alexa is not only biased towards the sites you mentioned (and of course all sites) but it is also biased towards users of InternetExplorer (which is the only browser its toolbar works with).

    Some of the sites I run have a 20-40% userbase of non-IE traffic, so right there you can write off all those users for improving my sites Alexa ranking…

    I would love to see Google release a quarterly top 1 million websites list, based on the same formula as pagerank but taking into account actual incoming traffic from the google toolbar.

  • Neil Patel

    Chris, we at Text Link Ads also feel that Alexa is not too reliable, but there are not too many other services that provide this kind of information. For this reason we feel that using multiple sources will help paint a more accurate picture of a website.

  • CarstenCumbrowski is smart. They have an affiliate program and pay $1.55 for each refered user who downloads and installs the toolbar.

    Did anybody play around with Trellians Popularity Index Rank?

    Also don’t forget NetCraft and (TrustGauge)

  • FreeKill

    Hi Neil,

    My apologies, I didn’t mean to single you out. I actually just applied to join so it was the first thing that came to mind.

    Really, many places use Alexa as a determinant for “site status” which sucks for any of us little guys but really, there is no alternative so how can you blame them?

  • Cvos SEO

    A better way to estimate a specific website’s popularity is to look at the number of backlinks, social bookmarks (such as delicious) Google PR, technorati links, Wikipedia references, number of cached pages, PPC competition, and number of pages a search for “company name” produces.

    Single source datasets are never reliable.

    Would you buy a house based solely on its valuation? A smart investor never would in a million years.

  • Mmoore

    Does anyone else not see the frailty of a ranking system based on a tool that is, by deafult, flagged as invasive and removed by SpyBot Search & Destroy?

    Approximately 80% of all internet users I know personally, run SpyBot and Ad-Aware at least every other day on default settings and eliminate all cookies and all link trackers… I suspect the ratio of overall Internet users who do this might not be that high but vast numbers of families are networking at home and installing protective software without bothering to read any instructions and allowing it to run the show… these folks “skew” things too – they throw off AdWord conversions and click charges, and cause lost affiliate income.

    It’s not entirely fair to blame Alexa’s skew on SEOs and Webmasters – some of us are not stupid and don’t want to artificially “enhance” our numbers because it nullifies our own research. I have Alexa installed on IE for purposes of accessing the stats on sites I research stats on, but I browse using Firefox.

  • osama

    u r right

  • buzziboy

    Great information, thanks for sharing it with us. i will try it out for my newly kids portal

  • johnson

    What I don’t really get is how some websites can jump so much in rank or lose a lot of rank in a few days or weeks. I read that you can make your alexa rank higher by visiting your own site with the alexa toolbar.