Reading Time: 6 minutes
What is AB Testing?
5 (100%) 1 votes

With the rising cost of traditional advertising, traffic acquisition has become an expensive activity. Making the most out of existing traffic is an effective alternative. So, what is AB testing and how does it work?

ab testing guide

AB Testing involves two versions of a single webpage. Version A is the currently used version (the ‘Control‘), while Version B is the modified page (the ‘Treatment‘). By running both pages simultaneously, their performance data can be easily compared. 

The modification of Version A should be based on a Hypothesis about how visitors use your website. This might relate to the design, the structure, or the content. By comparing the two versions (the Treatment and the Control) you can either prove or disprove your Hypothesis

Performing numerous AB tests is the best way to gain a real understanding of how a webpage’s design affects its performance. For large eCommerce companies, the process is continuous and involves many versions of each page.

ab testing definition

Testing variables is not new; some of the most familiar scientific ideas have been established by analysing the effect of variables through statistics. However, “statistical hypothesis testing,” in which a treatment is compared to a control, was established in the early twentieth century by statisticians.

Statistical hypothesis testing has one major advantage. It provides us with a way of calculating the statistical significance of an effect. Applied scientists such as Ronald Fisher and Jerzy Neyman used this technique in their biological and genetic experiments. Although they were involved in high-profile disagreements, together they helped to establish the principle of the Null Hypothesis

The use of statistical hypothesis for analysing marketing campaigns was a remarkably immediate application. In the world of advertising, early copywriters such as Claude Hopkins experimented with ways of testing public engagement. Hopkins used the return rate of promotional coupons, associated with different versions of advertising copy, to measure the impact of competing phrases. He described his techniques for testing copy in his Scientific Advertising (1923). 

Since the turn of the century, A/B testing has been used as a key resource for software providers, eCommerce platforms and internet services. The availability of vast sample sizes (provided by seemingly inexhaustible free traffic) and automated recording has led to the technique’s widespread adoption by major companies.

Writing in the Harvard Business Review, Ron Kohavi (Microsoft engineer) and Stefan Thomke (Business Administration Professor) noted the following examples:

  • In the year 2000, engineers working for Google ran a test to find the optimum number of results to display on a search engine results page. The answer (10 results per page) has remained consistent ever since.
  • In the early 2000s, Amazon discovered that moving credit card offers from its homepage to its shopping cart pages boosted revenue by tens of millions of dollars annually.
  • In 2009 an employee for Microsoft suggested a new way of opening links from the MSN homepage. 900 000 UK users were involved in the test, which directed browsers via a new tab, so that their session was not interrupted. The engagement of users (measured according to the number of clicks on the MSN homepage) increased by 8.9% and the practice has become a standard homepage tactic.
  • Similarly, in 2012, an engineer for Bing wrote a simple test for comparing two ways of displaying Ad headlines. The winning variant resulted in a 12% increase in revenue.

Today, companies such as Amazon and Booking.com each run over 10 000 A/B tests annually. Despite the difficulty of obtaining reliable data, and the relatively minor impact of most modifications, AB Testing is the most reliable way to improve website performance

In order to perform effective tests, website editors need three things: 

  • A hypothesis
  • A way of editing their site
  • A way of analysing their results

A website editor’s hypothesis is simply their idea for changing one element of a webpage in order to improve its performance. This might be the location of a call-to-action, the layout of a page, or even the colour of an add-to-cart button. 

AB Testing Software monitors and records the effect of the change on visitors’ behaviour. The software divides traffic between the ‘treatment‘ and the ‘control‘ and measures the different responses. The most sophisticated tools use algorithms to send more visitors to the best-performing version of a page. That way, businesses don’t lost out on customers whilst the test is running.

ab testing example

Once the website has received enough visits, the editor will end their experiment. However, there is another important step to make before the changes can be made permanent. Analysing the statistical significance of the experimental data is a crucial phase in the A/B Testing process.

AB Testing involves a single variable, with two versions of a page. Testing multiple versions of a page simultaneously is known as A/B/n Testing. Supposing a second variable is added, the page versions tested would be: AB, (N). Because this can involve any number of variables (depending on the available traffic).

a/b/n testing illustration

Multivariate testing works the same way as A/B testing, but tests more than one variation at a time, both separately and in combination. This gives information on how each individual Treatment works and how variations work together. 

multivariable ab testing illustration

The pages tested (A, B, C and D) will feature the following combinations of variables:

  • A – Y1 and X1 
  • B – Y1 and X2
  • C – Y2 and X1
  • D – Y2 and X2

Split testing is the same as AB testing, except the two pages, A and B, are assigned their own URLs. This makes the loading speed of the pages faster, and allows for more extensive changes. However, it is also a more complicated procedure.

A/B Testing has one major advantage over alternative ways of designing and redesigning a website: It is based on evidence! Whilst UX design, best-practice guidelines and customer journey analysis can provide hints and suggestions, A/B testing offers certainty.

CEOs and Marketing Directors prefer to base their decisions on data. Split Testing tells them how to achieve their goals:

  1. E-commerce websites use it to strengthen their conversion funnel
  2. Saas websites use it to improve their home page and enhance their sign-up process
  3. Lead generation websites use it to optimise their landing pages

The same process is also used to help redesign websites. In 2017, for example, British Airways launched a new website. However, before releasing the new design, they trialled new versions of each webpage with A/B Testing software. By the time the finished website was published, each page had been tested over several months and thousands of visitors.

Any element of a webpage can be tested by comparing the Control with a Treatment. Common features selected for testing include:

  • Titles – These are the first and most direct form of communication you will have with your visitor
  • Product Pages – The most beautiful shop in the world won’t make an ugly jumper sell
  • Copy – Tell a story, make a pitch, tell your visitor how you feel … words matter 
  • Prices – You can’t offer different prices to different customers, but you can change how they appear
  • Images – The way a viewer will relate to an image is impossible to predict, so data is essential
  • Colours – Google is famous for testing the colours on its website
  • Forms – Form completion rates vary far more than conversion rates, so optimization is key

Advanced tools allow you to test more complex elements, such as your site structure.

AB Testing take-away

AB testing is similar to the experimental process we are all familiar with. However, instead of simply observing the effect of a change on a variable, the object of AB testing is to achieve the biggest impact possible.

As with any other test, human error frequently interferes with otherwise convincing results. So, if you are considering performing AB tests on a website, make sure you avoid falling into the two most common mistakes: 

  1. The classic problem of confirming your own opinions.
  2. Trying to draw conclusions without having enough traffic, time or uplift (this is the most common AB testing mistake)

To make sure your tests are successful, follow the Convertize step-by-step guide to A/B testing

Stephen Courtney

by Stephen Courtney

Stephen is a digital marketer with a background in the history and theory of technology. He now writes about marketing and large technological systems.

Newsletter signup

Stephen is a copywriter and CRO fanatic. He writes about web psychology and marketing.

Subscribe to get his latest articles FREE