A/B Testing vs. Multivariate Testing vs. Multipage Testing

21st Jan 2019 –
Convert Experiences is a powerful CRO solution that you can use to power all your A/B testing experiments and discover the most effective CTA buttons, headline copies, hero images, color schemes, logo placements and more to use on your website.
The maths of multivariate testing:
Traffic was split equally in all the three versions (~ 33% each) to see which version got the most sales.
Multivariate testing or MVT is a way to test more than one variant against the control.
But thanks to the fact that you can make changes to clusters of elements for each variant, you can go from identifying a larger problem/conversion roadblock (which you do with A/B testing) to fine tuning the visitor experience for a particular page.

What Is A/B Testing

If they are feeling ambitious though, they can go in for funnel testing and actually play around with copy, a cart abandonment strategy and more.
When it comes to running CRO experiments, you can choose among four types of tests: A/B tests, split tests, multivariate tests, and multipage tests.
Convert allows for both conventional A/B/n tests and split URL tests through its Visual editor and the more robust code editor.
Companies usually use multivariate testing as an optimization technique to get more granular with their testing. Let’s get clear right off the bat. This is why the experimentation cycle is prolonged.
For example, a high bounce rate from a home page is generally indicative of the fact that visitors do not get the “value” the product or solution offers. A/B tests with different versions of a value proposition headline are common.

Split URL Testing

Group of elements to test: Element 1, Element 2.

With Convert, you can easily set up multipage tests or funnel experiments and see how different changes impact conversions across pages.
Multipage tests or funnel tests can be used to make high-level decisions across the entire sales journey.

Here the goal is to find the one that performs the best.
Element 2 variant B
Multivariate tests are extremely complicated owing to the inclusion of several elements over several variants. So both the traffic needs and the time to reach a conclusive outcome are compounded.

An A/B test example

So, for example, in its multivariate test, it could create multiple variations by changing the color of the CTA button, the hero image, and the CTA button copy and run it against the original to find a “more” winning combination.
Multivariate tests can be used to make more granular decisions. For example, if you have a high-performing homepage that you’d like to further optimize, you can use multivariate tests to experiment with more fundamentally different variations of a group of elements on your homepage.
So, if you want to test two radically different ideas, then you should go with split URL testing. For instance, if you’re looking to test two homepage designs, then this would be the right technique to use.
What it is:

• Version A (the control without a headline)
• Version B (the challenger with a headline)
• Version C (the challenger with a variation of the headline)

And when you test these copies (called the challengers) against the original one (called the control), you can see how the alterations impact the conversion rate.

A/B testing use cases

Element 1 Variant A
Or you can test how the presence/absence of recurring element(s), like security badges, can impact conversions across an entire funnel. This is classic or conventional multipage testing.
So let’s understand what these different optimization testing techniques are and how they compare with each other … and how you can be sure of choosing the right test each time you run an experiment.
You get the idea, right?

The pros of A/B testing

• A/B tests are simple to implement: Whether you test two very similar or two entirely dissimilar experiences, setting up an A/B test is quite straightforward. They usually aren’t too time or resource-intensive.
• A/B tests need lesser traffic (as compared to multivariate) tests to yield definitive results: If you’ve a website where you get enough traffic to send each variant about 10k visitors, you should be able to use A/B tests to find statistically significant results to make design, UI, or copy changes to your website. This is especially true when you run A/B tests with just the control and a challenger (or, say, two challengers) as you can still send good volumes of test traffic to each.
• A/B tests don’t need too long to run. Usually, A/B tests can yield statistically significant results in 2-3 weeks and don’t need to run for months and months on end.

The limitations of A/B testing

• A/B tests usually offer limited insights: A/B tests of similar experiences (where just one element is varied) usually run two versions. Which means, the most that a marketer can learn from the experiment is which of the three versions (1 original + 2 challengers) did the best. That’s all — insights about just three possibilities. If you want to test more variants (for example, if you want to test 5 versions of the CTA button), you’ll need enough traffic to produce statistically significant decisions even after splitting the traffic for that many variations.
• A/B tests are mostly only suitable for making high-level website decisions: Because A/B tests are great at giving objective answers as to which version is the best, they work great at taking high-level decisions. So you can decide upon the approach you want to take with your website or landing page’s design or UI or copy with A/B tests. BUT if you want to finetune the more granular elements of design, UI, or copy, then you’ll need to likely run several A/B tests, which can be time consuming and labor extensive.

A/B testing tools

An experiment can give definitive results only when it uses the right test format.
And unless you choose the right test for your experiment, making conclusive data-backed CRO decisions can get difficult.
When to use: 
Let’s create a hypothetical multivariate testing extension of the case study example from above.

Element 2 Variant A
Through A/B testing you can take high-level design and copy decisions.
So its experiment using a simple A/B test ended up with three versions:
Now, because this particular website is an online store, its most basic sales funnel is:
When to use: 
Now, because this eCommerce store has already used an A/B test to find a winning version of its flagship product page, it can now follow up with a round of multivariate testing to experiment with different combinations of the elements on the winning design to further improve its conversion rate.
Here’s the breakdown of a simple multivariate test:
These elements could be any graphic or text items from the page — for example, its CTA or headline copy or the hero image.
Variants of Element 2
When to use: 

Multivariate testing use cases

Again, let’s build a hypothetical extension to the same case study example from above.
At its core, A/B testing is the method optimizers use to identify the biggest roadblocks to conversion. You have to create a hypothesis that pinpoints the (purported) reason for low conversions and tests how changing/tweaking an element can mitigate the issue.

A multivariate test example/case study

This is the most common use case of A/B tests where you experiment with just one element and see how the change impacts the conversion rate.
Split URL testing it used to test totally different designs or versions of a page.
Think of it as plugging the smaller leaks after the deluge has been stemmed by A/B testing.

The pros of multivariate testing

• Multivariate tests are great at making micro-conversion decisions: With multivariate tests, you can experiment with many combinations of design, UI, and text changes. Which means you can take all the guesswork out of the process of “making tweaks” to your website for better conversion rates. You can actually validate even your most minute choices.
• Multivariate tests give deeper insights about how the elements play together: Because multivariate testing lets you test combinations of elements, you can see how your different combinations impact your users’ choices (rather than just testing stand-alone elements as in the case with A/B tests). With multivariate testing, the observed effect is always greater when there are several changes applied to a variant as opposed to only one change. This is called the interaction effect.

The limitations of multivariate testing

• Multivariate tests need a lot of traffic: If you look at the maths of multivariate testing, you’ll see that the number of versions to test gets huge depending on the number of elements to test and their variations. For example, if you’d like to use a multivariate test and test the different combinations of a CTA button (in three colors: one control and two challengers and three copies: one control and two challengers), then you’ll need to create 3*3, i.e., nine versions. Having enough traffic to split among so many versions is a challenge even for high-traffic websites.
• Multivariate tests can get difficult to plan: Planning a meaningful multivariate test is tricky because of the sheer number of combinations (of the different elements on your page) you can test. Choosing between combinations like a headline, a hero image, and the hero area CTA or the headline, USP, and the header can take a lot of effort. Also, testing too many elements in a multivariate test can skew the results.
• Multivariate testing isn’t suitable for taking definitive high-level decisions: Because multi-variate testing tests combinations of elements, you can’t always conclusively tell why a certain combination won. Which means taking a radical CRO decision is not always possible with multivariate testing. This reduces multivariate tests to powering just the more granular CRO decisions like testing placements/graphics/copy of the elements in the version you already know to work well.

Multivariate testing tools

In multivariate testing, you test more than one variant against the control (that you already know to perform well).

Multivariate Testing vs. Split Testing: Which Should You Use?

As a synopsis, this is what you should keep in mind:

What Is Multipage Testing & Funnel Testing

There is another form of testing that is often clubbed with A/B testing, but is fundamentally different from A/B tests.
With Convert Experiences, you can easily set up multivariate tests on your website and see how the different combinations of the elements impact your conversion rate.
In A/B testing, you test two very similar versions of the same page/ad/asset to find the best-performing variant.

Here goes.

Multipage testing and Funnel testing use cases

With A/B testing, you can add an element to a page or remove it or change it, and see how the addition, elimination, or change impacts the conversion rate of the page.
Multipage testing is a form of experimentation where you test changes to particular elements across multiple pages.
If you hypothesize that a certain element can make a positive impact on the conversion rate, you should go with this testing technique. For instance, you can use this optimization testing technique to see the impact of a change to your homepage’s CTA.

• You can run funnel tests to test different voices and tones on your pages — for instance, if you’re an anti-virus service provider, you could test a fear-inspiring copy in your sales funnel against your neutral or trust-inspiring copy — and see what messaging resonates the best with your visitors and gets the most sales.
• Design philosophies. You can experiment with different design theories and see which one works the best.
• Sales/support strategies. You can design a challenger sales funnel version where you offer live chat support on a few select pages of your funnel and see how the live chat enabled funnel compares to the regular sales funnel (without live chat support). Likewise, you can test a host of sales/support strategies and see how you can assist more conversions across your sales funnel.

A multipage test example/case study

Businesses that choose to invest in funnel testing yield tons of high-level insights.
Each of these tests has its own strengths and limitations.
What it is:
So, to run a multipage test, this store would need to create a variation for each of these pages and create a challenger funnel such as:
A/B testing has a very specific use case:
For example, you can test a personalized sales journey against a regular one.
SplitBase conducted its research and hypothesized that adding a benefit-driven headline to the landing page (which had no headline originally) could help rake in more sales.
Generally these versions are hosted at different URLs but the end goal is to boost conversions for a particular action, like say form fills or adds to cart.

The pros of multipage testing

• Multipage funnel testing helps plug leaks in the sales funnel: If your Google Analytics data shows major drop offs on certain pages of your sales funnel, you can set up a multipage test to find what can reduce them and get more conversions.
• Multipage tests can give deeper insights into user behavior: With multipage testing, you can identify the different elements of your control and challenger funnels that your users engage with the most. You can use such insights to take even more informed design, copy, and user experience decisions.

The cons of multipage testing

• Multipage experiments can last for months. Because B2B has long buying journeys (often stretched over months), B2B multipage tests can run for months on end.
• Gaining insights from a multipage test’s results can be tricky: If your challenger version experiments with too many variables (for example, if you experiment with both the design and copy in a challenger), you might not be able to tell if the design caused more conversions or the copy. Creating a bunch of challengers (for example, one for testing the design and one for testing the copy) will need you to split your traffic in three. Unless you get thousands of visitors to your website, you might not have enough traffic to get statistically significant results when you run too many such variations.

Multipage testing tools

A/B tests need about 10K visitors to each variant and around 1000 conversions to yield insights. But they are relatively quicker to conduct when compared to multivariate tests.

A/B testing vs. Multivariate testing vs. Multipage testing

A/B Testing

In A/B testing, you create one or more exact copies of the original page with the only non-identical elements being the elements to test.
In A/B testing, you create one or more exact copies of the original page with the only non-identical elements being the elements to test.
In A/B testing, you create one or more exact copies of the original page with the only non-identical elements being the elements to test.
Give Convert Experiences a full 15-day no-obligations free trial and rethink how testing can improve your KPIs.
There are two ways to go about this.
As you can see, an A/B testing experiment can very well become an A/B/C/D…n testing experiment.

Comparing A/B testing, Multivariate and Multipage Testing

In split testing the traffic a page receives is split down the middle and directed to two completely different versions of the same page in order to identify the version that performs better against the specified goal.
With multipage testing, you are at the liberty to gauge the impact of changes made on elements across a full buyer’s journey.

• A/B Tests – You generally test at least one significant element that has been identified as a conversion roadblock in your hypothesis across one other version (variant).
• Split Tests – Split tests focus on hosting two completely different pages and experiences for visitors on two separate URLs while evaluating them for the same goal of conversions. Through split tests you can test two completely different set-ups – like hosting platforms and themes against each other.
• Multivariate Tests or A/B/N Tests – MVTs are typically conducted on winners of A/B tests. In Multivariate testing you make changes to clusters of elements of a page, each unique or tweaked cluster becoming a variant or version of its own. It’s difficult to attribute higher conversions for a MVT variant to a particular element, but optimizers can test granular modifications to copy and design.
• Multipage Tests – Running a multipage test is similar to running multiple A/B/N tests together on a bunch of pages of the sales funnel. Alternatively, you can test entirely different sales funnel experiences (for example, sending half of the traffic to the control version Page 1> Page 2> Page 3 and sending the other half to the challenger version Page 1 variant> Page 2 variant > Page 3 variant with the challenger being quite dissimilar to the control.)

A/B testing is a way to test changes to one or more elements on a page.
Ease of Gaining Insights:
Check out the multivariate testing vs. split base testing (or A/B testing) comparison table down below to see when you should use either of the two techniques.