Information and advice for healthcare consumers.

Stuff we've learned

How SearchMetrics And Sistrix Got It Wrong, And Why

SearchMetrics OPI Graph For

SearchMetrics OPI Graph For

Just over a week ago Google’s Panda algorithm update was rolled out for all “English language Google users”. Given that our own traffic is largely English-language, and largely Google sourced, we paid particular attention to both our own traffic, which thankfully has increased since the Panda update, and any reports that were available about how other sites were being affected.

Two reports in particular got a lot of attention from bloggers and on Twitter. They were from SearchMetrics and Sistrix and they attempted to give some indication of the drop in “visibility” of particular websites after the algorithm was rolled out.

Two Problems With “Visibility”

The first problem with both of these reports is that they measure whether or not certain websites appear in the search results for a number of keywords that the companies are tracking. SearchMetrics for their part say that “Over 55 million domains and 25 million keywords are continuously monitored“. Sistrix say that their report is “based on a dataset of one million keywords“.

But what about the visibility the websites in question have for keywords these two companies aren’t tracking? How much of the websites’ visibility is invisible to SearchMetrics and Sistrix? The answer to these questions started to come quite quickly from companies who appeared on the “biggest loser” lists.

Demand Media, owners of issued a press release aimed at their investors and an accompanying blog post both decrying the inaccuracy of the reports. Doug Scott of went a step further and published pictures of his Google Analytics account to show how “we have lost none of our traffic“. A number of other companies have since raised their hands and said that they haven’t been affected nearly as badly as the loser lists suggest.

Out of interest, I looked back at our own organic traffic since January 1st and looked at how many different keywords were used to find our website. We have had over 1.4 million unique visitors to the site since the start of the year, and they used 1 million distinct keywords to find us. There is no way that either SearchMetrics or Sistrix could hope to measure our visibility in the search results accurately with that many different keywords in play. The only people who could release accurate data about any website’s visibility would be the search engines themselves, and they’re not likely to do so any time soon.

Visibility Does Not Equal Traffic

The second problem is that readers were confusing visibility losses with traffic losses, and SearchMetrics and Sistrix didn’t do a lot to correct this misconception at the time. In fact SearchMetrics mentioned “the statistical value of traffic distribution” and Sistrix mentioned “click-through rate on specific positions” as elements of their visibility indices, i.e. they were trying to calculate the websites’ traffic for the tracked keywords based on their search results positions.

As with our own example, many of the sites mentioned will have very high numbers of distinct keywords bringing in small amounts of traffic each, meaning that the SearchMetrics and Sistrix visibility indices bear little or no relation to the sites’ actual traffic. In fact, wary of some of the criticism coming their way SearchMetrics have now published a new blog post saying that “Searchmetrics OPI does not calculate the real traffic coming in to web pages“.

Why Publish A Visibility Index Based On Incomplete Data?

So why would these companies publish visibility loser lists knowing as they must how people would misinterpret them? The answer is clear: links and online mentions. A quick look at Yahoo Site Explorer shows that the SearchMetrics article has over 670 external links in, and the Sistrix article has over 150. SearchMetrics generated 111 comments on their post along with 225 ratings, and Sistrix managed 15 comments. There were also lots of Twitter mentions, although these are harder to measure because of the different URL shortening services used. [Figures correct as of 19th April 2011 @ 10am]

SearchMetrics and Sistrix both managed to hop on a rapidly moving story that was being closely monitored by some of the most-likely-to-link people in the world, the SEO community, and they succeeded brilliantly in creating classic link-bait. Their data sets were used to fill a temporary gap in knowledge and spread like wildfire. I even tweeted about the SearchMetrics report in relation to the loss reported for because it was so astonishing. What a pity then to find out that the data behind the reports couldn’t hope to tell the whole story in relation to the mentioned websites’ total visibility and traffic.

9 Responses to “How SearchMetrics And Sistrix Got It Wrong, And Why”

  1. Marcus says:

    Hi Philip,
    Visibility is an important part of an analysis of a domain. For every SEO it is necessary to know the performance of their sites and competitors in organic, paid and universal search. Sites with a big drop in visibility must have a structural problem, e.g. after a relaunch, a penalty or a change in Googles algorithm. An visibility is only addicted to keyword rankings and traffic from Google. If you have traffic from other sources or a lot of type-in traffic, than you don’t lose traffic if Google drops your site. But sites with a high percentage of Google traffic like ehow lose traffic if they lose visibility. You can see this also at

    And that is what Searchmetrics is doing. And your story about the linkbait isn’t true. All loser sites after the Panda update with a high percentage at Google traffic lost. Next example is

    That’s why we publish our index, because Google hit some sites really hard!

    • Philip Boyle says:

      Hi Marcus,

      Thanks for the quick reply. I understand how useful a measure of a site’s visibility would be if it could be comprehensive but as it stands both SearchMetrics and Sistrix can only measure some unknown percentage of any site’s visibility because you can only track a finite number of the keywords that might bring them traffic.

      Your 25 million keywords would only cover a handful of sites if you wanted to measure their visibility comprehensively. Seeing as you are tracking 55 million domains that means that in fact you are only tracking a relatively small number of keywords specific to any one of those domains, and most likely they are the high volume head as opposed to long tail keywords.

      Any site that lost ranking for these high volume keywords would see a big drop in its visibility as measured by SearchMetrics and Sistrix and yet may not see any significant drop in traffic if its traffic comes mainly from the long tail.

      In fact, the SearchMetrics blog post yesterday says as much:

      “Searchmetrics OPI does not calculate the real traffic coming in to web pages”

      “Search Engine visibility is not everything”

      “many of the sites that may now be experiencing reduced visibility as a result of the Panda update, may still continue to generate good traffic to their sites and continue to prosper.”

      I would also take issue with your statement that “Sites with a big drop in visibility must have a structural problem”. That will be true in some cases for sure, but for many sites that don’t depend on the keywords currently being tracked for visibility it is not.

      We can agree to disagree about whether the original post was link-bait, but the fact remains that visibility indices are only as useful as the percentage of the potential traffic they can measure for a site, and clearly in some cases that percentage is quite low.

      Finally, Alexa’s traffic data is patchy at best, and I while it might help show an overall trend, I would never depend on it to know what was going on.

  2. Uwe says:

    Hi Philip,

    For ages, Sistrix and Searchmetrics analysed search visibility for many websites in Germany. Here in Germany, it’s a kind of industry standard for most of all webmasters challenging Google results.

    It seems you and some others dont believe, that a well chosen sample of keywords in combination of search volume and ranking positions is able to calculate an organic search traffic trend for most of all relevant domains…

    But it’s quite simple to proof these results with third party metrics you trust. For example, check out the upcoming Nielsen or Comscore numbers for / and I’m sure you will get your confirmation.

    Next to Searchmetrics or Sistrix we are calculating a similar visibility score called “search engine reach”. In difference to all the others, we are not checking millions of keywords every week to calculate that index score. We are asking an user panel like Comscore or Nielsen or Alexa do so but purely broken down to organic search results.

    The panel is searching for all the multifarious long tail queries you mentioned above… and here are the results from SEO DIVER:

    -> in US:

    -> in UK:

    -> in UK:

    As you can see from the screenshots, we also recognized a huge decrease in search engine reach.

    For further details just drop me a line.

    • Philip Boyle says:

      Hi Uwe,

      Thanks for taking the time to comment. Part of the problem I have with companies claiming to measure visibility is they are happy to shout about their data when they get things right, as with etc, but they’re not so happy to address the claims of others who say that the visibility measures don’t match their own website’s traffic trend, as with and others.

      I fully believe that “a well chosen sample of keywords in combination [with] search volume and ranking positions is able to calculate an organic search traffic trend”. The problem is that the sample of keywords being used is obviously not well chosen or properly representative in relation to some of the websites that were included on the “biggest loser” lists.

      Your user panel, if sufficiently large, would offer better “real world” data including more long tail searches than either of the other visibility companies mentioned, but user panels are open to bias too, the most obvious of which are location, age and gender of the user.

      What I’d like to see is clarity and honesty when this kind of data is published. For instance, how may keywords were used to measure the visibility of each website, what were the demographics of the users in the panel measuring reach, and so on. That way, readers would be able to make their own minds up about how reliable the data is.

  3. fantomaster says:

    Got to agree with you here, Philip – this addresses a long standing peeve of mine. E.g. when clients keep asking “why aren’t you #1 for keyword XYZ” and I have to respond: “What makes you think we’re targeting that one at all?”
    Too many fond, unverified assumptions in place all the time which makes a lot of this data wrangling ludicrous and useless. You’re quite right: unless you have a comprehensive list of targeted keywords to test against, most of this stuff is no to avail.
    And let’s not forget serendipity (aka “the Long Tail” – more often than not, anyway…) as in key phrases and combinations that weren’t necessarily targeted specifically but simply popped into existence via the simple process of organic copy writing. Ever heard of the take that SEO “is not rocket science”? :)

  4. [...] Aufruhr sorgte Anfang der Woche ein Artikel auf, der besonders Searchmetrics und Sistrix nicht geschmeckt haben dürfte. Beide SEO-Tool-Hersteller [...]

  5. porter erwin says:

    Discount Vouchers seems to be your main evidence when there is far more evidence pointing to the analysis being helpful in understanding changes in Google’s algorithm. Sites like Pocket-lint & ItProPortal listed as big losers have both come forward and said they were hit by Panda. Discount Vouchers has also said that only 10% of its traffic comes from Google Serps. Why didn’t you include this because surely it affects reading its traffic before & after Panda?

    • Philip Boyle says:

      Hi Porter,

      Thanks for the comment. Don’t get me wrong – visibility is a very useful measure. It’s just that these lists of biggest losers are somewhat misleading, particularly in the cases of websites whose traffic is long tail.

      Unless a representative set of keywords is chosen for each website, the meaningfulness of any visibility score published is called into question for every site mentioned. How does the reader know which ones are more or less related to actual traffic (which is how most readers interpret them)?

      I’ve discussed with Marcus offline about how I’d like to see more clarity about what the data actually represents when it is published, and that way people can make up their own minds about how useful the information being presented is.

      Discount Vouchers was just the example I used because I had the links to articles about their traffic to hand. I haven’t seen the report that says Google only accounts for 10% of their traffic but I take your word for it.

      There were however other cases reported too including [BBC Report] and more in discussions on Webmaster World.

  6. Shaunda says:

    I got whacked by Google Panda personally aswell…

    I have recently been looking for ways around the Google Panda update and I’ve done a whole bunch of analysis on my own niche sites.
    Here are a couple of key points to take into account:

    1) minimize bouncerate on your websites
    2) delete/nofollow/noindex thin and poor quality pages of content
    3) boost Ctr in the SERPS (E.g use crazy call to actions and stuff on your page titles)
    4) provide more increased value on your site (boost users onpage time)
    5) interlink your web site with optimized anchor texts
    6) use a privacy policy, bulletpoints, paragraphs, subtiles, images, address/phone number, SSL, google maps location, contact us, about us etc etc (all “possible” extra quality elements you can think of)
    7) Get additional quality back links with effective anchor text variation.

    Building backlinks must be done in a very diverse and consistent fashion.

    Another thing, I am a beta tester for LinkALoha currently and i’m seeing GREAT results. I know they are going to open up a couple new spots shortly, you guys should check it out. Those guys are awesome


[Read More Stuff we've learned Posts]