One of the most mysterious areas in split testing seems to be – how to tell when a new ad or landing page has received enough traffic to know if it’s going to work or not. The quick answer is: A/B testing for statistical significance.
Today I’m going to do my best to explain that answer and show the results of a recent experiment I did that demonstrates what I’m talking about.
“He that can have Patience, can have what he will” – Benjamin Franklin
What is A/B Split Testing?
A/B Split testing is the practice of testing two or more versions of a creative element or elements against each other. By splitting the traffic into different groups and sending each group to different pages, you can see if new ideas will work better than old ones.
I previously wrote about A/A/B progressive split testing, a method of split testing that protects you from losing too much money.
Each split test should begin with a hypothesis.
For example: If you have a current landing page that is getting a 10% CTR and you need a 12% CTR to break even on the campaign you might try testing different Call to Action Button colors to see if more people click Red buttons than Blue ones.
In this case the hypothesis is: People will click Red more than Blue.
Using a split testing software like CPV Lab, you would add each of the landing pages to your campaign and give each one 50% of the traffic. That way half your traffic goes to the old page and half goes to the new page.
What is Statistical Significance?
From Wikipedia: Statistical significance is the probability that an effect is not due to just chance alone. It is an integral part of statistical hypothesis testing where it is used as an important value judgment. In statistics, a result is considered significant not because it is important or meaningful, but because it has been predicted as unlikely to have occurred by chance alone.
For example: After setting up your split test you may quickly find that your new landing page is getting a 25% CTR versus the old page’s 10% CTR. Then, over time, as more visitor data is gathered those results may balance out or completely reverse.
In that case the early 25% CTR was due to chance alone and was not significant.
Many online marketers make the mistake of making decisions too soon. By doing this they actually pause the better pages and slowly decrease their campaign’s performance over time.
How Long Does it Take?
How long does it take to get statistical significance? Well, that is a hard question to answer because like many things in life it changes on a case by case basis.
*Remember this: The higher the payout the longer test.
Example: If you have a campaign spending $100/day and are selling a product with a $100 payout, breaking even would mean you make 1 sale a day. It could take weeks or months to get significant data.
If however, your offer has a $1 payout, breaking even would bring you 100 sales a day so you could get significant data much faster.
A/B Testing for Statistical Significance
CPV Lab Case Study
I recently read an article that talked about increasing conversion rate by adding Urgency words after the CTA button text. The words “Now” and “Today” were recommended as being the best.
I decided to give it a try.
Using CPV Lab I setup a 5 landing page test. I used my control CTA and added their two recommendations (now and today) as well as two others that have worked well for me in the past “online” and “instantly”.
As you can see on the initial burst of traffic the landing page using the word “Now” did double the conversions of the control page. In the past, seeing a doubling of the conversion rate may have made me pause all other pages and call it a day… but these days I know better.
I ran it through the statistical significance calculator I use and got the following results.
According to the rules of significance a test is not over until one of the variations reaches a 95% confidence or higher. At this point I was just under 80% so I knew it was too early to decide.
After another day of traffic you can see that the landing page with the word “Now” on it was still looking really hot. It had gotten 5 more conversions than the control and looked like it was going to be a runaway winner. Each page had over 2000 views and over 350 clicks. (“Today” and “Online” were eliminated early).
At this point I thought I had found something really special and was going to really improve results.
Again, I had to hold off making a decision because even with the stellar performance of the “Now” page, I was only at a 74% confidence.
By day 5 of the test things had completely changed. My initial leader “Now” had fallen behind and was being destroyed by the control page (the original landing page).
After another few days of testing the control hit a 95% Confidence score, beating the living hell out of all the new versions.
I would have completely screwed my campaign if I had made a decision too soon.
What Happens When You Decide Too Soon
Has this ever happened to you?
You start a new campaign. It does well for the first few days and you start optimizing. Slowly, as time goes by performance dips and you just cannot seem to get it to make money anymore.
You know what may be happening?
Your rapid optimizations may actually be leaving you with sub-par landing pages that perform worse and worse with every change and every split test you do.
Why I Hate Most Split Testing Software
There are a lot of split testing softwares on the market. One of the most popular is called Visual Website Optimizer. The reason (I believe) it’s so popular is because they have done a very good job of designing it in a way that is simple, entertaining and addictive.
But I’m not sure it’s effective.
Looking at the screenshot you can see that the interface is slick. But when you see it in action it’s a thing of beauty. It’s more like a video game than a split-testing software.
Things light up, buttons change colors, levers and bars move around and when you find a 95% winner after just a few clicks – it makes you feel like a genius.
The problem is that entertainment aside, the results (for me) are rarely accurate. After several months of playing with VWO I decided to see what would happen if I let a test go past the point where VWO said I had a winner.
90% of the time the VWO declared winner did not actually perform best over time.
Does that mean VWO doesn’t work? No way, plenty of successful marketers use it and are making a killing. But for me – the entertainment value outweighs the results so it’s not a tool I use or recommend anymore.
Tools, Reference Links and Recommendations
I use three tools to split testing landing pages and calculate statistical significance. You’ve probably heard me talk about them before and that is because they work – and I will always use them.
- Crazy Egg. Crazy Egg is a heat mapping tool that sits on your landing page and records where your visitors click. It then shows you a “heat map” of hot zones on your page. On a new campaign I put Crazy Egg on the page, let some data pile up and see where people are clicking the most. Then, I optimize that exact area first. By doing this I maximize the chances that changes I make will have a massive impact on performance. Read my Crazy Egg review.
- abtester.com This is a free Statistical Significance calculator that I use to see if I have enough data to make decisions. I enter how many views and conversions the old page got versus the new pages and it tells me if I have a winner. In theory, if you reach a 95% significance, you have a 95% chance your new page is going to work better. It’s not fool proof but is very, very helpful.
- CPV Lab This is the industry standard for tracking campaigns and split testing landing pages. It’s a couple hundred bucks and is worth every dime. All of the screenshots of my live campaigns posted on this site, Facebook and Twitter are all from CPV Lab.
These three tools are more than enough to make sure you make smart decisions that are meaningful and help ensure you’re not driving your campaigns into the dirt.
CPV Lab Coupon Code
The guys at CPV Lab have given me a coupon code for my readers who want to try it. Click here to check it out and use CPV Lab coupon code “MALAN50” to save $50.
Split testing landing pages is one of the most fun parts of the affiliate marketing game. And if you can get good at it you can make just about any campaign profitable.
The most important factor in split testing is Statistical Significance. If you make decisions before the data is meaningful you can end up hurting your campaign, keeping the worst landing pages and ultimately making less money with every test you do.
Go slow, be patient and rely on longer-term results. Doing this will give you a better shot of continually increasing your ROI over time.
And that my friend, is the name of the game.
For more, watch my YouTube video: A/B Testing for Statistical Significance