There are three things that really irritate me about A/B testing. The first is where people fool themselves by drawing conclusions from too little data. The second is the myth that small changes frequently result in large improvements and the final one is when A/B tests are used to predict an actual percentage improvement when the data just isn’t there.

You Need a Lot of Data

Instructions for WhatClinic.com

The proposed improvement

We do a lot of A/B testing at WhatClinic.com and we like to think we know a little bit about the topic. We recently ran A/B test where we put a section of instructional text at the top right hand side of the page. After 11,000 tests and 400 conversions it clearly showed that the instructions made a 30% difference. It would have been so easy for us to stop there and pop open the champagne and boast about how changing one little thing improved our bottom line by 30%.

Things look Great - 30% Improvement

But we didn’t, we kept the test running, because experience has told us not to draw conclusions too quickly.  We let the test run on for another 90,000 people and 3,000 conversion and you know what. In the end it turns out that there was no substantial difference between the two. That’s right no difference.

The whole point of A/B testing is to learn. Learn what works and what doesn’t work. If you don’t run your tests over a large enough sample size then there is a good chance you are going to learn a fallacy. Not only won’t you be moving forward but you will actually be moving backwards and decreasing the value of your company.

Where did my 30% improvement go?

So what if you don’t have the traffic to do A/B tests? Well don’t do them. Do user testing. Get people in and ask them to use your product. You’re going to get a lot more information a lot faster and have a higher degree of confidence in the results.

Small Tweaks rarely makes Substantial Differences

I read about these all the time. You know the type of story – “I changed the colour of a button and increased conversion by 25%”. They read great and play into a pleasant dream that riches and fortunes are just a colour change away. However, in my experience small tweaks have never made a substantial difference to conversion.

It should come as no surprise to you that in order to substantially change user behaviour you need a substantial change to the site. This doesn’t mean that it never happens. However, I suspect that it happens rarely and the bulk of the time it is reported on blog and forums that it is the result of drawing conclusions from too little data or just plain old link baiting. Unfortunately the truth is normally all too boring.

A/B test don’t tell you how much better one page will be over another page

A really common misconception is to think that A/B testing can show you how much better one version of a page will perform over a different version of the page. IT CANNOT. A/B testing can only give you a confidence rate of whether one page is better than another and the observed historic improvement.

Highly advanced A/B testing can tell you a confidence rating of whether there will be a 5% improvement or a 10% improvement, etc, but it cannot tell you what the actual improvement will be. Too often people are fooled into thinking that just because they have observed a 30% improvement during the test that there will be a 30% improvement in the future. Whereas the actual results of the test is that version A has a 93% chance of being better than version B – note no prediction of how much better

Let me know of any examples you have where A/B test have first shown one thing then the other.  I know James Kennedy from voiceover Ireland has one on his blog here

Correction

It has been pointed out to me that the above example only shows a 20% improvement, not a 30% improvement.  Sorry for the mistake