At RevaHealth, we do A/B or multi-variant testing on our website frequently. We have found that this is the best way of maintaing a website that consistently performs better over time. What’s fascinating about this form of testing is how often gut feel and professional advice turn out to be wrong. In our experience, nothing beats getting the pages out there and testing them with real users.

A couple of weeks ago we decided to A/B test our conversion page.

6a00e54f09f406883401156f1b2ead970c-800wi
This page hadn’t been examined properly in quite a while and had kind of organically changed as functionality was added to the product. When we looked at it we were shocked by how bad it was. I was amazed that anyone had the patience to deal with it at all!
There were loads of problems with the form including:
  1. Lots of distractions. Forms that convert well typically don’t feature anthing that could distract the user from the job at hand. Amazon take this so far as to even mute their own branding.
  2. The ‘call to action’ button was in the wrong place. Ideally it should be on the left hand side, directly under the fields.
  3. There were too many fields. You should always ruthlessly cut out unnecessary fields. In this instance we didn’t really need to have the consumer repeat their email address and we didn’t need them to tell us what treatments they needed as they typically included this information in the description field.
So we redesigned the page and we came up with the following:

6a00e54f09f406883401157011fd8a970b-800wiIt was a much simpler form and because it was shorter we were able to put the labels directly above the fields and still fit the call to action above the fold. I was sure it would perform better (Phil wasn’t, but he is an argumentative sort!). So we put it out to test, splitting 50% of our traffic to the new form and leaving 50% of our traffic to the old form.

Results came back indicating that the old form was performing better. I was aghast and examined the page in all contexts to discover what was wrong. It was then that I noticed something that invalidated the whole test. Something that should have been obvious right from the start. Take a good look at the two forms and see if you can spot it (click on the screen shots and they will open to full screen).
On the right had side of the first form you will spot the clinic’s phone number. But it’s not on the new form. This means that when someone lands on the old form they can do three things:
  1. They can fill out the form.
  2. The can go back.
  3. They can look up the phone number of the clinic, call the clinic and never even fill out the form (this is not necessarily a bad thing for us).
However, with the new form they could only do two of these things. This difference totally invalidated the test.
The moral of the story is, when you are A/B testing make sure you are testing like with like. We have redesigned the new form so as to do so. I’ll post back here in a few days with results.