Reading Time: 7 minutes
What is A/B Testing?
5 (100%) 6 votes

What is A/B Testing?

A/B Testing involves two versions of a single webpage. Version A is the current version (the ‘Control’), while version B is the modified page (the ‘Treatment’). By running both pages simultaneously, you can easily see which one generates more sales or sign-ups. 

Running A/B tests is the only way to optimize a website with certainty. For the most well-known digital brands, testing is a constant process that guides their web development. However, since the introduction of tools like Optimizely and VWO in 2010, A/B testing has also become an important part of digital marketing.

what is A/B testing

Examples of A/B Testing in Digital Marketing

Since the turn of the century, A/B testing has become a key resource for SaaS, eCommerce and business websites. Because it is easy to sample and track a website’s visitors, most users never realise that they are part of a test. In fact, website testing has played a surprisingly important role in a number of historical events:

  • 2000: engineers working for Google ran a test to find the optimum number of results to display on a search engine results page. The answer (10 results per page) has remained relatively consistent since then.
  • 2008: the presidential campaign for Barack Obama tested an early campaign donor page, identifying a combination of CTA copy and images that produced 40% more clicks. Similar tests were responsible for a reported $75 million in extra campaign donations.
  • 2009: an employee for Microsoft designed a type of link that would open a webpage in a new tab. The number of clicks on the MSN homepage increased by 8.9% and “open in a new tab” links have since become a standard homepage tactic.

Today, companies like Amazon and Booking.com each run thousands of tests annually. Despite the relatively minor impact of most tests, they remain the most reliable way to improve website performance.

How Does A/B Testing Work?

In order to perform effective tests, you need a “hypothesis”, a way to edit your site and a tool to record the results. Your hypothesis is simply your idea about how to improve your webpage. This might be changing the location of a call-to-action, the layout of a page, or even the colour of a button. 

what is a b testing

A/B testing software monitors and records the effect of the changes on your visitors’ behaviour. It divides traffic between the ‘treatment’ and the ‘control’ and measures the different responses. The most sophisticated tools even send more visitors to the best-performing page. That way, you don’t lost out on customers whilst your test is running.

ab a b testing sampling

Once your site has received enough visits, your software will declare a winner. However, there is another important step to make before the changes can be made permanent. Analysing the statistical significance of your data is a crucial part of the A/B testing process.

A/B/n vs Multivariate – What is the difference?

A/B testing involves a single variable (ie. a call-to-action button) with two different versions. When a test involves multiple changes, it can be either an “A/B/n test” or a “Multivariate Test”. Unlike an A/B/n test, Multivariate (MVT) testing shows you how different variables work together and which ones have the biggest influence on your conversions.

What Is A/B/n Testing?

Testing multiple versions of a particular element is known as A/B/n Testing. Supposing you want to try three different colours of button, the page versions would be AB and C. Because you can add any number of different versions, this kind of test becomes A, B and “n“.

what is a b n testing

What Is Multivariate Testing?

Multivariate testing works the same way, but compares more than one variable both separately and in combination. That gives you information on how each individual version works and how different variations work together. 

what is a b multivariate testing

For example, testing alternative versions for two separate elements (a call-to-action button “X” and a header image “Y”) would mean comparing four combinations. That would give you the following combinations of variables:

  • A – Y1 and X1 
  • B – Y2 and X1
  • C – Y1 and X2
  • D – Y2 and X2

Multivariate testing requires an exceptionally large sample size, so it is only really possible for the largest websites.

What is Split Testing?

Split testing is the same as A/B testing, except the two pages, A and B, are assigned their own URLs. This makes the loading speed of the pages faster, and allows for more extensive changes. However, it can also be a more complicated process and there is a larger possibility of contaminated data.

When Did A/B Testing Start?

Testing variables is not new. Scientific discoveries have always been based on experiments and data. However, “statistical hypothesis testing,” in which a treatment is compared to a control, was established in the early 20th-century.

Statistical hypothesis testing has one major advantage: it gives us a way to calculate the size of an effect and the likelihood that it occurred by chance. Scientists such as Ronald Fisher and Jerzy Neyman used the technique in their experiments. Together they helped to create concepts like the Null Hypothesis.

In the world of marketing, copywriters such as Claude Hopkins applied these new concepts to advertising. Hopkins used the return rate of promotional coupons to measure the impact of different campaigns. He described his technique in a book called Scientific Advertising (1923).

Why is A/B Testing So Popular in 2020?

Testing has one major advantage over alternative ways of optimizing a website: it is based on real users. Whilst UX design, best-practice guidelines and customer journey analysis can provide hints and suggestions, real-world testing offers certainty.

  1. E-commerce websites use it to strengthen their conversion funnel
  2. Saas websites use it to improve their home page and enhance their sign-up process
  3. Lead generation websites use it to optimize their landing pages

The same process is also used to help redesign websites. In 2017, for example, British Airways launched a new website. However, before releasing the new design, they trialled new versions of each webpage. By the time the finished website was published, each page had been tested over several months and thousands of visitors.

What Should You Test? Examples of Successful Tests

You can test any aspect of a webpage. The most common things to test are the most visible elements. Advanced tools allow you to test more complex elements, such as your website’s structure.

  • Titles – H1 text is crucial for both SEO and conversions.
  • Notifications – Helpful notifications can have a significant impact at key stages of your sales funnel.
  • Images – The way a viewer will relate to an image is impossible to predict, so testing is very useful.
  • Forms – Form completion rates vary far more than conversion rates, so form optimization is key.

The most important aspect of your website depends on your aims and your industry. These are some of the most successful tests from the past decade.

1. A/B Testing Titles and Subtitles – Highrise

One of the most famous A/B tests from the past 10 years was a simple lead generation sign-up page for the CRM software Highrise. The test compared four different title and subtitle combinations, including the original. 

a b testing example

Although Highrise used a small sample for their test, they were confident with the outcome of their experiment. The page title which gave the length of a free trial and explained how easy it was to sign up converted 30% more frequently than the original.

2. A/B Testing Notifications – SportChek

Displaying shipping details early in the checkout process helps to reduce abandoned baskets. In 2019, the eCommerce platform SportCheck experimented with ways to advertise their free shipping policy on product and basket pages. However, the results were disappointing. Adding a notification to their product pages increased sales by a tiny margin, and the test had only reached 55% Confidence. 

free delivery example a b test

Exploring the data more carefully, the Director of Experimentation discovered that sales from the test pages had begun to decline on 29 July. On that date, the store’s free shipping policy had been updated so that it only applied to orders over $75. Before the update, the notification had increased sales by 6.56% with 96% Confidence. 

3. A/B Testing Images – Zagg

One of the reasons why it is important to test your content is that our assumptions are often wrong. For example, most agencies would recommend using a video as the first image on a product page. In this A/B test, conducted by the mobile accessory retailer Zagg, a 360-degree image outperformed the product video by 11.9&.

a b test product images

The test measured the average revenue per customer, so that low-value sales could not affect the outcome. The results were significant to a Confidence Level of 95.4%.

4. A/B Testing Forms – Thomas Printworks

This A/B test is another example of how best-practice guides can be wrong. Optimisation specialists usually insist that shorter forms, with fewer fields, are likely to convert more effectively. In fact, this test from the car customisation company Thomas Printworks found that a longer page with more specific questions received far more completed forms.

webform a b test

The original page was completed by 2.2% of visitors, whilst the longer form converted 5.28% of the time. The increase represented a 140% uplift, at a Confidence Level of 90%.

A/B Testing In 30 Seconds (Animated Video)

As with any other kind of test, human error frequently interferes with otherwise convincing results. So, if you are considering performing A/B tests on a website, make sure you avoid falling into the two most common mistakes: 

  1. The classic problem of confirming your own opinions.
  2. Trying to draw conclusions without having enough traffic, time or uplift (this is the most common A/B testing mistake)

Whilst most A/B testing tools will do all the calculations for you, it is still worth taking some time to get familiar with A/B testing statistics.

Download The Guide For Free

Download the complete Introduction to A/B testing here. The Guide is completely free and comes with a jargon buster, industry examples and a history of A/B testing

Your subscription could not be saved. Please try again.
Thank you, we’ve now emailed you the PDF.
convertize what is ab testing
Stephen Courtney

by Stephen Courtney

Stephen is a digital marketer with a background in the history and theory of technology. He now writes about marketing and large technological systems.