The Death of A/B Split Testing

death-of-split-testing

It’s official… A/B Split Testing is Dead.

I hope you’re ready. Because today I am killing old school A/B Split Testing. But don’t worry – I’m also going to teach you what to do instead. 

Proper split testing is one of the most important skills any online marketer needs to have. Because if you split test correctly – you can make almost any campaign profitable. And by using the method that I’m about to teach you – you won’t have risk a lot of money to do it.

So please read carefully and get ready to learn.

In this post I will show you:

  • How I learned to split test
  • Why A/B split testing is Dead
  • Why my method works better
  • How to set it up in your tracking software

No landing page yet? Read: How to Launch Your First Campaign

I Use CPVLab

For this example we’ll be using CPVLab, my tracking software of choice.

One of the main reasons I use CPVLab is because it simplifies the split testing process and allows me to control the exact percentage of visitors who see my new page (which is key to the whole thing).

If you want to use CPVLab click here and use the promo code “MALAN50″ to save $50. Offer expires on March 1st so if you want it, now is the time to get it.

Note: If you’re using Prosper202, don’t leave. There are also instructions here for you too.

My First Split Test

In late 2008 I launched my first attempt using paid traffic. It was an Adwords PPC campaign on the search network with a $50/day budget.

In earlier years I’d been a web designer so I had 100s of sites under my belt. I figured the landing page part of it would be a breeze.

I spent two days crafting the perfect design. I obsessed over the details. It was beautiful. As I activated my campaign and started the traffic I knew I would succeed… I mean who could resist this amazing design.

The traffic started coming in. I got really excited… Then –

Nothing.

After my first day’s $50 budget was spent I looked at the stats in the affiliate network and realized that I’d made no sales. I then logged into my tracking software and saw that my landing page had gotten zero clicks.

Not a single click.

Confused – I went to an online marketing forum and wrote a post about what was happening. I received a single response. At first I considered it an insult.. then I realized it was the best advice anyone had ever given me.

“99% of the time the problem is not the campaign, it is the affiliate”

What he was saying was that the campaign was not failing because of a problem with the campaign. It was failing because I was the affiliate. And if I could fix “me” I could fix the campaign.

“Fixing Me” meant I needed to step up my game and split test like a super affiliate to find a winning ad > landing page > offer combination.

My super-designed landing page sucked. And I had to admit that. I was pissed. But I now understood more about how the game works.

So I Started Split Testing Like F#cking Crazy

I began rotating a fury of landing pages to the traffic until I found a design that got a few clicks. A few design changes later and it was getting a lot more clicks. Then with a few more tweaks I started getting conversions.

Then just for fun I downloaded the homepage of a blog I read, added my own content and tested it. It looked awful – but it started getting clicks AND conversions like crazy.

2008-splittest

In about a week I went from 0% Click Thru Rate (CTR) and zero sales to a healthy 20% CTR and sales coming in. I did this by testing multiple pages against each other and then making changes, based on the data I found.

You May Want To Read: How I Figure Out What to Split Test

The 3 Rules of Split Testing:

From that first experience I realized that testing is about the most important thing I can do. And from that day forward I have 2-3 landing pages rotating at all times.

Here are the 3 rules I live by:

  1. If you sell something online you should almost always use a landing page
  2. If you using a landing page you should always split test multiple versions
  3. If you’re not split testing right now – you need to start, immediately

What is Split Testing?

In affiliate marketing “Split Testing” a landing page is the process of comparing the performance of two landing pages to see if one converts better than the other. You compare them by “Splitting” your traffic between the pages.

For example: Your homepage has a yellow “Buy Now” button and has a click through rate (CTR) or 10% and a conversion rate (CVR) of 1%. You need to get your CTR to 15% in order to make a profit at 1% CVR. So – you split test to find something that will get you more clicks.

The Hypothesis

Each split test you do should be based on a hypothesis (a proposed explanation for a phenomenon). In this case – the hypothesis is that an orange buy now button could get more clicks and conversions than a yellow button.

Why Test Colors?

It has been well documented how much colors can impact the way people behave on your website. This isn’t theory this is a scientific fact.

For example: In a recent study, Scientists at the University of British Columbia studied more than 600 people. They found that the color red makes people more cautious and attentive to details while blue makes us more creative and open to new ideas.

split-test-colors
The test pages from the University study
Source: University of British ColumbiaNPR.org, Science DailyUVM.edu

So for this test this is our hypothesis:

Hypothesis: Changing the color of the Buy Now button will increase the number of clicks and conversion that our landing page gets.

The A/B Split Test

The most common method of split testing is the A/B method. To test the hypothesis (that orange might beat yellow) with the A/B method we split test two pages.

landingpage-yellow-orange2

To create our test page we copy the existing page (Page A) and replace the yellow buy now button with the orange button and save it as a separate page (Page B) and upload it to the website.

Using CPVLab we split our web traffic into two groups, “A” and “B”. I also add the start date to the LP Name so I know when the test began.

split-5050
Normal A/B Test in CPVLab

One group is shown Page A and the other group is shown Page B. After running the test we can watch and see if Page B creates more clicks and conversions than Page A.

If it does – Page B wins the test and becomes the new landing page to beat and Page A gets thrown in the trash.

Benefits of A/B Split Testing

When you start a new campaign using a brand new landing page you will most likely lose some money. I’d say 90% of my campaigns lose money on day one.

The golden 10% that either break even or make money on their first day are a beautiful thing, but definitely do not happen every time.

So my only job is to transform the campaign from losing money to making a profit. I do this job by testing new things and improving performance.

This is the art of being an affiliate. It’s why some people can do it and some can’t.

“If you combine creativity with a relentless testing schedule – you can make almost any campaign profitable”. (Tweet This)

Things to Split Test

There are many things you can split test on a landing page. In fact – you can split test every element on a page if you have a lot of traffic. Here are a few of the things that I’ve seen make the most impact on page performance.

  • Headlines
    • Probably the most impacting thing you change. Visitors read this first and decide whether or not to look at the rest of your page.
  • Images
    • The first image a visitor sees can change the way they feel about your page.
    • A photo of the author helps build trust with the reader (if they like the photo)
  • Call To Action buttons/links
    • Very important area to test. This is the text and button or link that goes to the offer, normally at the bottom of the page.
    • After reading your headline, looking at your image and scanning your copy – this is what they’re going to click (hopefully). Test it relentlessly.
  • Copy
  • Captions under images
    • You’d be shocked at the difference captions (with links) can make on a page.
  • Font Size
    • If your readers are age 40+ try increasing font size, as we get older it gets harder and harder to read tiny text on the screen
  • Background colors
    • Colors are powerful. Red causes fear, Blue causes peace.

Downsides of A/B Split Testing

The problem with traditional A/B Testing is that if your new page (Page B) doesn’t perform as well as your original page (Page A) you will lose money on the 50% of your visitors that are shown Page B.

revenue_split-test2
How you lose money while split testing

If Page A has a CTR of 10% and Page B has a CTR of 3% and your landing page gets 5000 visitors a day – you lose 350 click thru’s every day of your test.

Some tests require multiple days – or even weeks to get statistically significant results. So if Page B is performing poorly – you can lose a ton of cash.

Different Types of Split Testing

The type of split testing I do normally depends on the type of campaign I’m running and the value of each conversion – or cost per action (CPA).

To make decisions you need clicks and sales (data) and depending on the type of offer, getting statistically significant data can take a few minutes or a few days (or weeks).

For example: An auto insurance zip submit campaign (visitors enter their zip and click Submit) pays $2 per conversion brings in 1000s of conversions per day. I can make decisions pretty quickly. I get a nice, large pool of data to go by very fast.

A campaign for Rolex watches (visitors pay $2500 for a watch with a credit card) that pays $750 per conversion may only bring in 1 conversion per day. So I have to wait weeks to get the amount of data the insurance campaign gives me in 10 minutes.

Note: I recommend new affiliates start with low payout offers so you can get do tests like this without spending thousands of dollars and spending tons of time.

My Method: A/A/B Progressive Split Testing

aab-split-bulbs

No matter what type of offer you’re promoting and regardless of payout – A/A/B Progressive Split Testing is a method I suggest you try. It’s called A/A/B Progressive because we’re using 3 landing pages and the test builds slowly over time.

It takes a little longer to get results – but it also protects you from losing money on 50% of your visitors while your split test is running.

What is A/A/B Progressive Split Testing?

aab-split-testing2

During a normal A/B split test you’d put Landing Page A up against Landing Page B, run 50% of your traffic to each page and see which one performs better.

During that test you are risking half of your traffic on the new page because it may or may not perform well.

To eliminate the risk – I give just 10% of my traffic to the new page. This way if it performs poorly, only a small percentage of my traffic is wasted.

Why not just give Landing Page A 90% of the traffic and Landing Page B 10% of the traffic and see how things go?

The Problem:

The 90% vs 10% A/B split test is flawed because the landing pages are getting different amounts of traffic at different times of day, which means the data could be inaccurate.

The Solution:

My goal is to give each page the exact same amount of traffic at the exact same time of the day for the exact same amount of time. This is the kind of test I can trust. But I don’t want to risk half my money by doing a 50/50 split.

How to Set UP A/A/B Progressive Split Test in CPVLab

Here’s what it looks like:
– Landing Page A (1) gets 80% of the traffic
– Landing Page A (2) gets 10% of the traffic
– Landing Page B gets 10% of the traffic

split-801010
An 80/10/10 split test in CPVLab

Benefits
The first benefit in this setup is that Landing Page B (the unproven test page) is only shown to 10% of my visitors. So if it performs horribly I’m only risking 10% of my profits (versus 50% in normal A/B tests).

split-testing-risk
A/A/B Progressive Split Testing Reduces Risk

The second benefit is that my split test Page A (2) versus Page B receives the exact same amount of traffic for the exact same period of time. This setup ensures that the data I get is as close to accurate as possible.

Potential Drawbacks

A drawback to this split testing style is that it takes a bit longer to get your data. When you launch an A/B test with each page getting 50% of the traffic you get to see how the page performs faster.

But losing money fast has never been my goal. I prefer to be patient and test slowly, because it is always more profitable in the long run.

Adjusting Your Split Test Over Time

Now that you have traffic coming in you’ll keep an eye on Page B and see how it does. If it appears that Landing Page B is going to do well (has great CTR or CVR) you will give the split test more traffic by raising it’s percentage in CPVLab to 20%.

To see if your page is going to do well – use this statistical significance calculator.

But remember – you want each page in the test to receive the exact same amount of traffic so when you increase page B to 20% you have to increase Page A (2) to 20% and decrease Page A (2) to 60% at the same time.

The new test looks like this:
– Landing Page A (1) 60%
– Landing Page A (2) 20%
– Landing Page B 20%

split-602020
60/20/20 Split Test from CPVLab

This way you can slowly increase the page, based on performance to speed up your test. Each page of the actual split test gets the same amount of traffic giving you more accurate data which the only important thing here.

*If Landing Page B is performing poorly, leave it at 10% even if it means your test takes more time. I’d rather test slowly than lose 50% of my revenue every day. If Page B never improves, pause it and create another page to test.

The Eventual A/B Split Test

If the new landing page seems to be performing as well or better than the original page you can eliminate Landing Page A (1) and move to a 50/50 split of the two pages.

You now know that Landing Page B is worthy of 50% of your traffic and will most likely not lose you a trunk full of money.

This looks like:
– Landing Page A (2) 50%
– Landing Page B 50%

split-05050

That is the art of the A/A/B Progressive Split Testing. That is how you get to a 50/50 split test that won’t lose your money. This is the skill I wanted to teach you today.

A/A/B Progressive Split Testing on Prosper 202

If you are using Prosper202 and want to do this you will need to set up the pages manually in your index.php rotator file.

*I have not used Prosper202 for years and it may have been updated. But last time I used it, this is how I accomplished the goal.

To achieve an 80/10/10 split you’d need to add your Landing Page A (1) to the script 8 times and add Landing Page A (2) and Landing Page B to the rotator once each.

This looks like:
– Landing Page A (1)
– Landing Page A (1)
– Landing Page A (1)
– Landing Page A (1)
– Landing Page A (1)
– Landing Page A (1)
– Landing Page A (1)
– Landing Page A (2)
– Landing Page B

screen-202
Image: An 80/10/10 Split Test in Prosper202

You would then do the progressive adjustments by deleting or adding page versions to the list. This is much more hassle and is one of the reasons I prefer paying for CPVLab.

Is A/A/B Progressive Split Testing Right For You?

In short – yes. I recommend this for every split test you do. I don’t see why anyone would do a normal A/B test unless they don’t mind losing some cash on the test, which is fine for super affiliates, but a bad idea if you don’t have unlimited funds.

No matter what style of landing page split testing you use I can’t stress enough the importance of split testing on a regular basis.

If A/A/B Progressive Split Testing isn’t for you – find something that works and do it all the time. Car salesman have the phrase “ABC” which stands for “Always Be Closing”. In Affiliate Marketing our phrase is “ABT” or “Always Be Testing”.

Split testing ads, landing pages and offers is the quickest way to take a campaign from loser to winner.

The Wrap Up

Today I’ve shown you a method of split testing that I use to improve my campaigns. It’s called A/A/B Progressive Split Testing because instead of testing two landing pages against each other (A/B) you test three and progress over time.

  1. A (original at 80%)
  2. A (original version 2 at 10%)
  3. B (new page at 10%)

By using A/A/B Progressive Split Testing I can:

  • Protect my profits while testing
  • Get more accurate data during the test
  • Eventually move to an A/B test that I know won’t lose money

I also gave you a solid list of things that I split test:

  • Headlines
  • Images
  • Call to Action areas
  • Copy
  • Captions
  • Font Sizes
  • Background colors

If you’re not testing these things now – then get started. You could be leaving major amounts of money on the table.

Take Action

If you want to start split testing your landing pages I highly recommend CPVLab as the tracker to use. Use promo code MALAN50 and save $50. (Offer valid until March 1st)

Click Here to Get CPVLab now

 

But if you’re brand new and need a free system – Prosper202 can do the job. Just know that it is rarely updated and not supported.

Either way, get one and start split testing today.

Also: Get my free eBook ’10 Decisions’ about how I basically split tested my entire life

In Conclusion

The art of split testing is probably the most important skill an online marketer can attain. By mastering this skill – you’ll be able to take campaigns that lose money (90% of all campaigns) and turn them into a money makers.

Start with the headline and move down the list.

*Tip for new affiliates: If you have a smaller daily budget – don’t test more than one thing at a time or your test will take weeks to produce any real results.

Whether or not you choose to use A/A/B Progressive Split Testing as your testing method – you should Always Be Testing (ABT). Improving a campaign’s performance is your only job – so be good at it.

Question: Are you currently Split Testing? If so what method or system are you using?

  • ONERank

    Great ideas Malan!!! Huge fan of CPVLab and that split test workflow is genius. Keep it coming!

    • http://malandarras.com/ Malan Darras

      Glad you enjoyed it. CPVLab is the jam.

      Do you have a human name? Feels weird talking to a logo haha.

  • Nikola Cvrtnjak

    Your method is practically same as mine. But I have a question for you about display advertising and split testing. How do you split test banners? I was thinking about that lately, and my method looks something like this. I pick an angle with one design/headline/body/CTA and test 10 different images just to get high CTR banner (don’t expect conversions, if I get some, great).. After I get the image I test with different headlines and designs and so on. How do you do it :D? I was thinking about changing that to testing different banner designs first or different headlines, and later on to search high CTR images. It’s not good to spread too much on banenr testing as well cause each banner needs to collect some solid data as well.

    Anyway. Great post Malan! 🙂

    • http://malandarras.com/ Malan Darras

      I’m relentless with banners.

      #1) I come up with 1-5 angles with images and headlines for each and do a short test
      #2) I take the best angle and create 20-30 headlines and keep the same image
      #3) I test 3-10 at a time looking for insane ad CTR + landing page CTR (shows interest level)
      #4) I take the best headline and make multiple versions of it and test again
      #5) After headlines I test images
      #6) After images I test landing pages

      #boom

      • Nikola Cvrtnjak

        tnx bro 🙂

      • Justin Lucas

        The banner split testing process has been one of my sticking points lately and this helped me a ton. Gonna try this out right away. Thanks, Malan!

        • http://malandarras.com/ Malan Darras

          I shoulda made it an eBook. 6 bullet points for $27 haha

      • bigaffdreams

        I can’t believe you just shared that. Great Karma is headed your way.

        • http://malandarras.com/ Malan Darras

          feel free to steal it. it works.

      • Ken

        hey Malan, thanks for this! Just a quick one: once you’ve found the winner you will keep split testing right? Let’s say after a while the numbers are no longer great. Then how do you find out what needs to be improved(or it’s the time to dump this campaign)? is it the angle? banner? lander? There’re SO MANY moving parts here…can you see where I come from? Thanks again for the great stuff and God bless you.

        • http://malandarras.com/ Malan Darras

          I’m am always split testing. 6-8 months later I am still split testing. It never ends. Diminishing returns is a myth, you might not be able to improve something for a while, then BOOM you improve your ROI again.

          I start with the part of the process that gets the most views.

          So like this:
          – Banner ad Image
          – Banner ad Title
          – Banner ad Copy
          – Landing Page

  • Marco

    You suppose that B is worse than A. So it would be OK to give B less impressions, but eben better Not to give B any impressions at all.
    If from your experience B is more often than not better than A, then its a mistake Not to share the traffic evenly, do you get my Point?

    Good idea to have separate groups to have the Traffic Split evenly. I have the Impression that CPV Lab does Not Split the traffic evenly when you introduce new LP After the campaign was already Running…

    • http://malandarras.com/ Malan Darras

      Marco – I think even the best landing page designers are guessing. I never know how my traffic is going to react to changes. So to safeguard my $$ I start the test slowly.

      And you are right. CPVLab tends to try to “catch up” the new landing page when you introduce one to a campaign that is already running. It will assign all of the traffic to the new LP until they’re even. That’s another reason I do it this way.

      I’m going to add that to the post. So thanks for bringing it up.

      • Marco

        Ok I understand.
        Thanks for confirming that “catch up” thing in CPV Lab. I’ll keep this in mind when adding new variations. Also a good tip from you was to add a date to the LP name in CPV Lab.

    • cmdeal

      You are right. That is why this method is flawed.

      In order to do it properly you need to run a proper Bayesian multiple armed bandit model, where the traffic split is adjusted dynamically based on the conversion of each page being tested.

  • http://www.leanvertising.com/ Jason Brown

    I’ve been using Optimizely for a few years to do all my LP split tests at 80%-20%.

    Most affiliates never think about cutting down their wasteful B side test results. Generally, I will rotate/test 4-5 different LPs out the gate at the 80/20 method I use and then pick the winner. All my B tests off the winner will also be 80/20 ( for element split-testing ).

    With that method, I always have 1 URL I send traffic to and Optimizely will record my changes, tests, handle rotations, etc itself.

    • http://malandarras.com/ Malan Darras

      Hey Jason – Same here. when i test more than one new lander I’ll split it up a little differently but still in an even percentage. For example if I have 3 new landers to test the split looks like:

      60% Page A
      10% Page A
      10% New Page 1
      10% New Page 2
      10% New Page 3

  • Eric

    Malan, thanks for giving golden information about A/B testing. This has opened up a new vertical for doing my split test. Cheers man

  • Kaldera

    A/B testing is not going anywhere.

    It certainly is not dying anytime soon.

    What you are describing is a very simplistic version of a basic multiple bandit model. This is from 1952, and it has a much narrower range of applications than standard A/B testing.

    If you don’t understand when a multiple bandit model is applicable, what sorts of probability distributions it assumes, and how to interpret the results, the decisions you make will be as bad as the people on Wickedfire who make decisions on campaigns based on $10 of spend and a sample size of 5 clicks.

    • cmdeal

      A multi armed bandit model can be useful in certain cases, but just arbitrarily splitting traffic by some predetermined static ratio as the author suggests is definitely not the right way to do this.

      In order to apply a multi armed bandit model in a web conversion context, the traffic split needs to be adjusted dynamically based on the conversion of each page being tested. The correct ratio will change over time, but it can (and usually will) be very far from 80-10-10.

      Setting a predetermined ratio at the outset can actually result in you LOSING a lot of money that a correct multi armed bandit model can avoid.

      • http://malandarras.com/ Malan Darras

        cmdeal – If CPVLab had the option to do dynamic adjustments based on conversions I’d use that. But it doesn’t so I use this + a few calculators and make it work. Last time I checked – I’m doing pretty well with it.

        Maybe we all get together and submit a feature request for what you and Kaldera have brought up – I’m game. Til then – it’s A/A/B Progressive Split Testing alllll day.

    • http://malandarras.com/ Malan Darras

      Thanks for the input Kaldera. Your E=mc2 is strong

  • http://blogprofitmedia.com/ Andy Baker

    I prefer to keep making radically different designs from scratch, and keep a low enough number in rotation that you get results fast. Then swap one out for another. Repeat for 10-20 designs, then tweak from there, staring with the most prominent copy first.

    • http://malandarras.com/ Malan Darras

      Andy – I love throwing in totally different designs in two places. The beginning (to find the angle/design to pursue) and then right after I think I’ve perfected the ad or landing page.

      I put myself in my competitor’s shoes and try to destroy my existing page. Sometimes I win, sometimes I lose. But i’m always fighting. haha

      • http://blogprofitmedia.com/ Andy Baker

        I still want an ad creative API that will auto generate relevant designs and headlines based on previous market winners, get manual approval then split test them and repeat. Probably a few years out though 🙁

        • http://malandarras.com/ Malan Darras

          Sign me up when that API is ready haha – I think that could be automated via outsourcing or employees now though. would just have to include humans

  • Robert

    Can you show us an Ad that is working great for you right now?

    • http://malandarras.com/ Malan Darras

      Hey Robert – I don’t post my personal ads. But some advice is:
      1. make them ugly
      2. Make people say to themselves “what is going on here?”
      3. Don’t give everything away in the title (leave them hanging)

  • Vasco

    Great Post Malan!! good stuff!

    • http://malandarras.com/ Malan Darras

      thanks Vasco.

  • Fred Page

    Is there an API that will enable DRTV traffic to a single URL be evenly split between three landing pages to test conversion rates?

  • Jon Tam

    Hey Malan, how would you do this in terms of weights in Voluum?

    • http://malandarras.com/ Malan Darras

      Jon with Voluum you can duplicate the A Lander with 1-click and give it a unique name, then do an A/Av2/B test.

      Weights would be 80/10/10 or you could do 800/100/100 or 8/1/1 if you want.

  • Mr.Leex

    Thank you Malan, this is really great testing strategy for those who are limited on budget like me. I found your blog recently and want to say your blog are packed with golden articles, Thank you very much for your writing!

    • http://malandarras.com/ Malan Darras

      thanks Xiao!

  • Gilberto Hernandez

    Hey Malan,

    Great post mate, took lots of notes from it and will be applying those when doing media buys : )

    I have one question for you, I’m focusing with PPV right now, do you apply the same methodology for PPV?

    Usually what I do is, for example, I create two landers, see which one is getting more conversions and from there I turn it into my control piece where I split test headlines, images, copy, CTA wording and colors, audio wording/voice, etc

    Every time I find a winner, I make it my control piece and try to beat it with a new split test until I have a winner design and then split test against a whole different page just to see how it goes.

    I would like to know your insights about PPV and split testing if it is possible : )

    • http://malandarras.com/ Malan Darras

      i would do the same thing for PPV

  • Nick Mascetta

    Great Post Malan,

    As a side note, is there any reason you prefer CPVlabs over Voluum?

    • http://malandarras.com/ Malan Darras

      i actually use Voluum now

  • Greg

    Thanks for a great post! Question: how would you go about testing ads on FB? Often I am not sure whether I should focus on artwork or copy…

    • http://malandarras.com/ Malan Darras

      First test angles – in other words test multiple “general ideas” for the campaign overall. Each angle has it’s own image, headline and ad copy.

      Then take your best angle and break it down and test in this order:

      1st. Images
      2nd. Headlines
      3rd. Ad copy

  • Love Raj Singh

    Thanks for the great post. Gonna try the A/A/B split testing. Guess the 80-20 rule works everywhere 😛

  • fouad

    Hey Malan, thank you with this great article,
    1) I was confused with this way of A/A/B or 80/10/10
    you said in the graph that the risk is 10%(the LPB), you supposed that the first LP A is a good one, but what’s happen if the first LP A is a bad one you will lose 90% of money instead of 50% in the first approach A/B ?!
    2) my second question is that how you split list of things that you mentioned above, can you explain in details the method that you use ?
    thanks in advance 🙂