Multivariate Testing: What it Is, Why You Need It

Friday, someone told me it’s hard to convince folks they need to use multivariate testing in their internet marketing campaigns.
Once my mouth closed, I decided to write this post.

What Multivariate Testing Is

If you already know, you can skip down to why you need it, below.
I can’t define it without twisting my own brain in a knot, so I’m going to use an example, instead.
Say I have a landing page. On this page, I want to test the headline, the body copy, and maybe how I link to the product sample:

You can see the actual page here. You can even hire us if you want. Pretty please?

That’s three variables. I want to find the combination that gets me the most sales. So, I’m testing two other headlines, two other body copy versions, and one other version of the sample pages:

and

I can split my visitors randomly into 3 groups and send them to each page, then pick the winner.
Problem is, that’s not a complete test. I don’t just want to test these three pages.
I want to test every possible combination of the variables on these three pages, then pick the best combination of variables, even if it’s not represented by one of my examples.
That’s where multivariate testing comes in. With one of the many multivariate testing tools out there, I can create a page template, then insert each of these variables, providing every possible combination.
More important, a well-written testing algorithm can tell me which ‘recipe’ will work best, even if that combination of variables never appeared on one page in the test.
So, multivariate testing lets me compare more than variables. It lets me compare combinations of variables, quickly and efficiently, and then use some mathematical wizardry to find the best possible page configuration.
Neat!

Why You Need Multivariate Testing

Internet marketing and multivariate testing were made for each other. You can’t use this method with magazines, television or other traditional media (not without going broke and insane, anyway). But online, you can display a different page layout, different copy, etc. to every visitor if you want to. It’s the perfect environment for sophisticated testing.
You need multivariate testing! You need it because:

1. It can end arguments. Ever spent 4 weeks going back-and-forth with a client/designer about which photo to use on a landing page? Tell ‘em you’ll test your professional shot against the one they took of their 3-year-old niece and let the algorithms sort it out. You’ll get your page live faster, and end the argument sooner.
2. Multivariate testing reduces uncertainty. Done right, it can help you zoom in on the most effective page layout.
3. Multivariate testing saves money. It reduces design cycles (see #1 above). It improves conversion rates (see #4 below). It avoids some of the catastrophic failures you might otherwise see, especially in landing page design.
4. It improves conversion rates. Well, duh. But think about it: Say you get 1,000 conversions a month now, and those earn you $20 each (a total of $20,000). You do a multivariate testing run and improve your conversion rate 10%. Meh, you say. Big deal. But that’s 100 more customers, worth another $2,000. That’s worth something, I’m sure. If you’re a bigger organization, multiply that by 100x or so. You get the idea.

Multivariate testing adds a little science to the art of internet marketing. It also brings incremental improvements to your campaigns that add up fast.
You need it as part of your strategy.
Other posts about testing:
Double e-mail open rates with this simple test
Internet marketing case study: Testing Works
10 Great internet marketing excuses, and what they mean

If you want to learn a bit more, check out 3 Steps to Make a Good Multivariate Test. Full disclosure: We’re a Widemile Partner, and I think they’re great.