Information and advice for healthcare consumers.

Stuff we've learned

How SearchMetrics And Sistrix Got It Wrong, And Why

SearchMetrics OPI Graph For

SearchMetrics OPI Graph For

Just over a week ago Google’s Panda algorithm update was rolled out for all “English language Google users”. Given that our own traffic is largely English-language, and largely Google sourced, we paid particular attention to both our own traffic, which thankfully has increased since the Panda update, and any reports that were available about how other sites were being affected.

Two reports in particular got a lot of attention from bloggers and on Twitter. They were from SearchMetrics and Sistrix and they attempted to give some indication of the drop in “visibility” of particular websites after the algorithm was rolled out.

Two Problems With “Visibility”

The first problem with both of these reports is that they measure whether or not certain websites appear in the search results for a number of keywords that the companies are tracking. SearchMetrics for their part say that “Over 55 million domains and 25 million keywords are continuously monitored“. Sistrix say that their report is “based on a dataset of one million keywords“.

But what about the visibility the websites in question have for keywords these two companies aren’t tracking? How much of the websites’ visibility is invisible to SearchMetrics and Sistrix? The answer to these questions started to come quite quickly from companies who appeared on the “biggest loser” lists.

Demand Media, owners of issued a press release aimed at their investors and an accompanying blog post both decrying the inaccuracy of the reports.┬áDoug Scott of went a step further and published pictures of his Google Analytics account to show how “we have lost none of our traffic“. A number of other companies have since raised their hands and said that they haven’t been affected nearly as badly as the loser lists suggest.

Out of interest, I looked back at our own organic traffic since January 1st and looked at how many different keywords were used to find our website. We have had over 1.4 million unique visitors to the site since the start of the year, and they used 1 million distinct keywords to find us. There is no way that either SearchMetrics or Sistrix could hope to measure our visibility in the search results accurately with that many different keywords in play. The only people who could release accurate data about any website’s visibility would be the search engines themselves, and they’re not likely to do so any time soon.

Visibility Does Not Equal Traffic

The second problem is that readers were confusing visibility losses with traffic losses, and SearchMetrics and Sistrix didn’t do a lot to correct this misconception at the time. In fact SearchMetrics mentioned “the statistical value of traffic distribution” and Sistrix mentioned “click-through rate on specific positions” as elements of their visibility indices, i.e. they were trying to calculate the websites’ traffic for the tracked keywords based on their search results positions.

As with our own example, many of the sites mentioned will have very high numbers of distinct keywords bringing in small amounts of traffic each, meaning that the SearchMetrics and Sistrix visibility indices bear little or no relation to the sites’ actual traffic. In fact, wary of some of the criticism coming their way SearchMetrics have now published a new blog post saying that “Searchmetrics OPI does not calculate the real traffic coming in to web pages“.

Why Publish A Visibility Index Based On Incomplete Data?

So why would these companies publish visibility loser lists knowing as they must how people would misinterpret them? The answer is clear: links and online mentions. A quick look at Yahoo Site Explorer shows that the SearchMetrics article has over 670 external links in, and the Sistrix article has over 150. SearchMetrics generated 111 comments on their post along with 225 ratings, and Sistrix managed 15 comments. There were also lots of Twitter mentions, although these are harder to measure because of the different URL shortening services used. [Figures correct as of 19th April 2011 @ 10am]

SearchMetrics and Sistrix both managed to hop on a rapidly moving story that was being closely monitored by some of the most-likely-to-link people in the world, the SEO community, and they succeeded brilliantly in creating classic link-bait. Their data sets were used to fill a temporary gap in knowledge and spread like wildfire. I even tweeted about the SearchMetrics report in relation to the loss reported for because it was so astonishing. What a pity then to find out that the data behind the reports couldn’t hope to tell the whole story in relation to the mentioned websites’ total visibility and traffic.