Reading Time: 10 minutes
The Complete Guide To A/B Testing & Split Testing (2020)
4.5 (90%) 154 votes

A/B testing lets you to optimize your website with evidence, rather than guesswork. This Complete Guide to A/B Testing and Split Testing contains:

  • A framework for A/B testing
  • A survey of the best A/B testing tools
  • A test Significance Calculator
  • Answers to the most frequently-asked questions about statistics, page loading speed and SEO 

It will help you decide if A/B testing is right for your website and show you how to do your first test.

A/B testing is always based around a goal. For most people, the goal is to increase their conversion rate. As long as you have a measurable goal, you can collect data from your users. However, you also need to have some other ingredients to conduct a successful test.

  • Traffic – You can’t test your website without a large enough test-sample
  • Hypothesis – You need some ideas about what changes to make to your original page
  • A/B testing Tool – A software tool to allocate your traffic and collect your results

Low Traffic is one of the major obstacles faced by first-time testers. In our article on A/B testing sample size, we outline the minimum traffic required to run successful tests.

There are different strategies for A/B testing, but most of them follow a familiar pattern. This is the 5-step plan we use at Convertize:

ab testing strategy

An effective optimization strategy is built on continuous testing. At the end of each testing cycle, it is important to revisit and evaluate your experiments.

Step 1 – Analysis

Before you’ve even begun to think about what changes you are going to test, you need to analyse the original version of your webpage. Google Analytics is a useful tool, because it tells you how visitors are using your site. There are also free CRO tools, such as heat maps and mouse-trackers, that you can use to study visitor behaviour.

how to do a/b testing

By examining this data, and finding weaknesses in your Funnel, you can identify what needs to be changed. A/B testing should always start with a hypothesis.

Step 2 – Creating Your Hypothesis

This is the fun part! You have to put your neck on the line and find a way to change Version A in order to achieve your goals.

So, what are you going to change? Your call-to-action buttons, your copy, the colours or structure of your sections? Does your page need a facelift or full cosmetic surgery? A good hypothesis needs to be clear, well-defined and based on your data.

how to do a/b testing

Step 3 – Designing Your Experiment

It is important to be precise about the settings for your experiment. Before launching a test, you need to decide on:

  • Your goal – In order for an A/B testing platform to compare the performance of version A and version B, you need to decide which action to measure. Most of the time, this is just the URL of a “thank you” page (following a purchase). You should use a “thank you” page URL, rather than a checkout, because it guarantees that you will only count real purchases. Occasionally, you might want to set a different kind of goal. Clicking on a particular button or visiting another page on your site are examples of “micro-conversions”. 

how to do a/b testing

  • Which pages to target – Testing each product page one by one would take forever, but targeting the wrong pages will ruin your data. Like with search engines, you can define the limits of your test in terms of: “URL contains”, “URL ends with” or “URL equals.”
  • How you want your traffic to be split – Some software comes with a “multi-armed bandit” algorithm that directs your traffic to the best-performing version of your webpage. This has two advantages. Firstly, it can help to reduce the time taken to achieve significant results. Secondly, it means conversions are not lost by sending valuable traffic to a less-optimal page.

how do do a/b testing

Step 4 – Running Your Experiment

Running an experiment is like sitting in the passenger seat. No matter how much you want to tweak, adjust and alter the process, you have to let the driver take control. However, there is one important decision you have to make: when to end the test.

Before starting your test, you should decide on the “confidence level” you expect to reach. A 95% confidence level is the standard for most agencies.

how to do a/b testing

If there is no variation between A and B, it will take a long time to get statistically significant results. In this case, it might be better to test a more substantial change.

Step 5 – The Interpretation

Even if you achieve an uplift with statistical significance, it is still a good idea to change things gradually. For example, version B might lead visitors to make a purchase more frequently – but it might also reduce the average amount people spend. It may be that you only showed version B to a segment of your customers. In that case, the next step would be to try it on the other segments.  

Buying an A/B testing tool is a good long-term investment. It allows your team to make better decisions and should help to create a culture of optimization in your company. However, most tools don’t suit every company. To find the right tool for your company, you need to ask yourself some questions…

If your team doesn’t have front-end development skills or programming knowledge, you need a tool with a simple drag-and-drop editor. If your team doesn’t have statistical expertise, you need a tool that has built-in statistics. Even experienced testers can be bogged down by the knotty details. So, if you don’t feel like wading through raw data, you need a tool that will do it for you.

2. What Kind of Volume Do You Have?

To put it bluntly: does your traffic match your ambition? To optimize a website through A/B testing you need between 10-100,000 visitors a month on each page you want to test. For multivariate testing (MVT), you’ll need many times that amount. 

3. Will You Need Other Tools?

To optimize your site effectively, you may need more than one tool. For example, you might think about using:

  • A heat map and scroll map 
  • A visitor survey popup 
  • A customer journey platform

An A/B testing tool is unlikely to combine all of these features, so you may need to set aside some of your budget for additional software.

Asking the right questions before you subscribe to a tool could save you a lot of money. It can also help you find the best A/B testing tool for your business.

The demand for A/B testing has led to a wide selection of tools designed to make the testing process as smooth as possible. However, not all testing tools work the same way, and most are aimed at executive customers (with executive prices). 

We recently conducted a survey of the 26 best A/B testing tools on the market in 2020. Very few of the tools available combined a user-friendly interface with the flexibility required for effective testing. 

best ab testing tools

These are 10 of the most popular AB testing tools available in 2020:

  1. Convertize A tool designed with marketing professionals and mid-sized companies in mind. It is a robust system and comes with expert support. The software offers intuitive editing, unique speed and safety features, and a library of neuromarketing tactics.
  2. Optimizely An executive tool for major eCommerce platforms. It provides advanced testing options. Optimizely supports multivariate testing and was built to manage large traffic volume.
  3. VWO – Visual Website Optimizer – Another popular tool with marketing agencies and large multi-national companies. Along with A/B testing, customers have access to a full suite of additional analytics (such as heatmaps).
  4. AB Tasty Originally built for medium-sized enterprises, the tool has been repositioned as an executive solution. AB Tasty now specialises in re-marketing features.
  5. Google Optimize Google’s free AB testing system. Optimize 360  provides a paid-for service that can test up to 10 variations of a page. The tool can be integrated with Google Analytics.
  6. Kameleoon An expensive option aimed at medium-sized companies. It is based around the use of AI and machine learning and has a focus on personalisation.
  7. Convert Positioned as the alternative to Optimizely. The tool has been rebranded for an executive audience but is nonetheless cheaper than its famous rival. It is one of the few executive tools to offer expert support.
  8. Omniconvert Targeted at small and medium enterprises. It is built to integrate with customer segmentation and personalisation. The software offers affordable multivariate testing.
  9. Adobe Target  One of the oldest A/B testing solutions, Omniture Test & Target, was absorbed into Adobe’s marketing suite. Adobe provides an executive service for major businesses (HSBC, for example). It is the most expensive, and most comprehensive, A/B testing package.
  10. FreshMarketer Zarget was purchased by Freshworks in 2017 and renamed. The tool runs within a Chrome extension and has an impressive list of features for such an affordable solution.

There is no standard time for an A/B test because a test is only considered reliable when the results are significant. Until then, it is dangerous to draw any conclusions (even from seemingly clear data). Statisticians are wary of a phenomena called Regression to the Mean.

  • Regression to the mean – When seemingly clear results become less pronounced as the sample size increases. If the variation between A and B appears significant to begin with, but regresses to a more moderate difference, then the initial results were probably the result of outlying phenomena. In this case, the variation will become less pronounced the longer your test continues.

In order to reach significance, your test requires sufficient Statistical Power. This is determined by the Effect Size and your Sample Size.

  • Sample Size – Before starting an A/B test, you must calculate the sample size needed. Your sample is composed of visitors to your website, so your test duration is directly related to the amount of traffic your site receives. In some cases it might be sensible not to run an A/B test because the volume of traffic available on the site (or the page tested) is not high enough.
  • Effect Size – This is the change caused by your variable. In the case of A/B testing it is measured in terms of conversion rate. A dramatic Uplift in conversions on Version B of your page would constitute a large Effect. The bigger the difference between versions, the more likely you are to reach statistical significance.
  • Statistical power – The chance that your experiment will detect an effect, if the effect exists. There are two significant factors that determine the statistical power of your test: the magnitude of the effect your test creates and the number of visitors your site receives.

These factors combine to give your test a degree of Representativeness (or, statistical significance). This is the likelihood that your results demonstrate a real effect.

Calculating statistical significance is an important step in any experiment. There are two main approaches to calculating significance: Bayesian and Frequentist. They are not simply alternative methods, but actually reflect different interpretations of probability. 

A Quick Guide To Statistics: Bayesian vs Frequentist Approaches

The Frequentist approach examines the number of times an event occurs in a volume of tests. The result is a statement only about frequency in a given sample.

The Bayesian approach starts with an estimate of a real-world effect and updates this as data is accumulated. The result is a new estimate of the real-world effect and a number describing how much it can be trusted.

ab testing significance

When calculating the statistical significance of an A/B test, both approaches contribute important information. A/B testing software often combines the two approaches in a single statistics package. Using your experimental data, the software will tell you the relative uplift observed between A and B and the likelihood that this is a result of the changes you have made.

You can use the Convertize significance calculator here


The method used by Convertize for analysing statistical significance is a hybrid approach combining elements of Frequentist and Bayesian statistics.

Google clarified its position regarding A/B Testing in an article published on its blog. The important points to remember are:

  • Use Canonical Content Tags: Search engines find it difficult to rank content when it appears in two places (“duplicate content”). As a result, web crawlers penalise duplicate content and reduce its SERP ranking. When two URLs displaying alternate versions of a page are live (during A/B tests, for example) it is important to specify which of them should be ranked. This is done by attaching a rel=canonical tag to the alternative (“B”) version of your page, directing web crawlers to your preferred version.
  • Do Not Use Cloaking: In order to avoid penalties for duplicate content, some early A/B testers resorted to blocking Google’s site crawlers on one version of a page. However, this technique can lead to SEO penalties. Showing one version of content to humans and another to Google’s site indexers is against Google’s rules. It is important not to exclude Googlebot (by editing a page’s robots.txt file) whilst conducting A/B tests.
  • Use 302 redirects: Redirecting traffic is central to A/B testing. However, a 301 redirect can trick Google into thinking that an “A” page is old content. In order to avoid this, traffic should be redirected using a 302 link (which indicates a temporary redirect).

By following these guidelines, you can make sure that your tests have no negative SEO impact on your site. If you are using A/B testing software, these steps will be followed automatically. All major A/B testing tools follow these guidelines and use javascripts in such a way that testing will not influence your SEO.

A/B testing software can reduce loading speed due to the way in which it hosts competing versions of a page. A testing tool can create scenarios in two ways: 

  • From the client’s side (front-end)
  • Using server-side scripts.

Server-side This form of A/B testing is faster and more secure. However, it is also expensive and more complicated to implement.

Client-side This approach uses Javascript code that makes the changes directly in the visitor’s browser. Client-side testing can create a loading delay of a few fractions of a second. The loading delay traditionally associated with client-side software is known as the “flickering effect”.

Most A/B testing software operates on a Client-side basis. This is to make editing a site as easy as possible. In order to reduce the impact of testing on a page, the best A/B testing solutions have found ways to speed up page loading.

The Complete Guide To A/B Testing 2020

Providing your site has enough traffic, A/B testing is an essential technique for any eCommerce business’s marketing team. Not only will it provide unexpected insights about your customers and your site, it will allow you to market your business with certainty. 

For websites with lower traffic volumes, there are alternative ways to optimize your conversion rates. For example, you might begin by exploring our 2020 guides to CRO and neuromarketing.

    Philippe Aimé

    by Philippe Aimé

    Philippe is the CEO of Convertize. He created his first website in 1998 and has spent the past 20 years finding ways to make digital marketing more persuasive. He now heads a team of CRO consultants.

    Newsletter signup

    Philippe is the CEO of Convertize. He created his first website in 1998 and has spent the past 20 years finding ways to make digital marketing more persuasive. He now heads a team of CRO consultants.

    Subscribe to get his latest articles FREE