Featuring The biggest names:
A/B Testing involves two versions of a single webpage. Version A is the currently used version (the ‘Control’), while Version B is the modified page (the ‘Treatment’). By running both pages simultaneously, you can easily compare their performance.
The modification of Version A should be based on a Hypothesis about how visitors use your website. This might relate to the design, the structure, or the content. By comparing the two versions (the Treatment and the Control) you can either prove or disprove your Hypothesis.
In order to measure the performance of your pages, the first thing to decide on is a “KPI” to measure. This could be the number of sales (“true” conversions), sign-ups (a form of “micro-conversion“), or another action. Whatever you choose, it will be the measure that determines which version of your webpage is performing best.
Running numerous A/B tests is the best way to gain a real understanding of how a webpage’s design affects its performance. For large eCommerce companies, the process is continuous and involves many versions of each page.
For a complete introduction to A/B testing, see: What is A/B testing?
How Do You Do
A/B testing is always based around a goal. For most people, the goal is to increase their conversion rate. As long as you have a measurable goal, you can collect data from your users. However, you also need to have some other ingredients to run a successful test.
Traffic – You can’t test your website without a large enough test-sample
Hypothesis – You need some ideas about what changes to make to your original page
A/B testing Tool – A software tool to allocate your traffic and collect your results
Low Traffic is one of the most common obstacles faced by first-time testers.
There are different strategies for A/B testing, but most of them are continuous and follow an “iterative” (step-by-step) pattern.
Step 1 – Analysis
Before you’ve even begun to think about what changes you are going to test, you need to analyse the original page. Google Analytics is a useful tool, because it can tell you exactly what is happening on your website. Alongside this, dedicated CRO tools can tell you why it is happening. By examining the data, and finding weaknesses in your “Conversion Funnel”, you can identify exactly what needs to be changed.
Once you have completed the analysis stage, you should be able to make a prediction about what kind of edits will improve your site. This prediction is your A/B testing hypothesis.
Step 2 – Hypothesis
This is the fun part! You have to put your neck on the line and find a way to change the “A” version of your webpage in order to achieve your goals. The new page (your “B” version) should be exactly the same except for a single alteration. The best hypotheses are clear, controlled and based on your data.
Step 3 – Design
It is important to be precise about the settings for your experiment. Before launching a test, you need to decide on:
- Your goal
- The metrics you will use to track it
- Which pages you want to target
- Whether your test is one-tailed or two-tailed
- How you want your traffic to be split
A/B testing is all about getting the most reliable results as quickly as possible – but you don’t want to sacrifice conversions whilst you test.
Step 4 – Experiment
Before starting your test, you should decide on the “Confidence Level” you expect to reach. A 95% Confidence Level is the standard for most tests. This means that in 19 out of 20 cases, any effect you observe will be due to a real effect and not random variation.
Choosing a higher Confidence Level will make your tests last longer and reduce the Power of your experiment (you will have more chance of missing an effect if it exists). However, it will also reduce the chance that you produce a “false positive”.
Step 5 – Interpretation
Even if you achieve an uplift with statistical significance, it is still a good idea to change things gradually and to keep checking your key metrics. This is because any change to one part of your website can have an unexpected impact on another part. For example, version B might lead visitors to make a purchase more frequently – but it might also reduce the average amount people spend.
Four years ago, the market for A/B testing tools was divided between two options: Optimizely and VWO. Since then, the number of A/B testing tools has increased rapidly. AB Tasty arrived, with a set of advanced analytics and, in 2017, Google launched Optimize and Optimize 360. In 2020 there are around 50 well-known options for A/B testing software.
A simple, user-friendly A/B testing tool for marketers
For a full survey of the most popular A/B testing tools, see: A/B Testing Tools (2020)
How To Choose The Best A/B Testing
Tool For Your Business
Subscribing to an A/B testing tool is a good long-term investment. It will allow your team to make better decisions and encourages a culture of experimentation and data-based decision making within your company. However, no single solution suits every company. To find the right option for your business, you need to ask yourself some questions…
1. What Skills Do You Have?
If your team doesn’t have front-end development skills or programming knowledge, you need a tool with a simple drag-and-drop editor.
Similarly, if your team doesn’t have statistical expertise, you need a tool that has built-in statistics. Even experienced testers can be bogged down by the knotty details. So, if you don’t feel like wading through raw data, you need a tool that will do it for you.
2. What Kind of Volume Do You Have?
To put it bluntly: does your traffic match your ambition? To optimise a website through A/B testing, you need between 10-100,000 visitors a month on each page you want to test. For multivariate testing (MVT), you’ll need many times that amount.
3. Will You Need Other Tools?
To optimize your site effectively, you may need more than one tool. For example, you might think about using:
- A heat map and scroll map tool
- On-page visitor surveys
- A customer journey mapping tool
An A/B testing platform is unlikely to combine all of these features, so you may need to set aside some of your budget for additional software.
Asking the right questions before you subscribe to a solution could save you a lot of money and will help you to find the best A/B testing tool for your business.
For a list of important things to consider, see: 10 Questions To Choose The Best A/B Testing Tool For Your Business
How Do You Get Statistical
Significance in A/B Testing?
For any A/B test, the most important concept for interpreting your results is Statistical Significance. This is the probability that the difference between your A and B page’s conversion rates is due to real changes in your visitors’ behaviour. Unless you have achieved statistical significance, there is a strong possibility that your results are due to chance.
Your Confidence Level is the minimum level of probability you are willing to accept that your results are NOT due to chance. It is often set at 95% (which means that any effect you find will be real in 19 out of 20 cases).
You can easily check if your test results are significant by using the calculator and instructions in this guide: AB Test Significance
Or, for a full guide to A/B testing statistics (including Variance, Bayesian and Frequentist statistics), see: AB Testing Statistics
AB Testing Sample Size
Finding a way to get Uplift is not easy, and achieving statistical significance is downright difficult. That’s why you need to think about your AB testing sample size before you launch an experiment.
To give you a sense of the sample size needed to run AB tests on your website, we have created a simple chart showing how many visitors your website will need to produce significant test results.
1. Fear Factor zone
With less than 10,000 visitors a month, AB testing will be very unreliable. That’s because with such a small sample, it would be necessary to improve your conversion rate by more than 30% in order to identify a clear “winning” variation. Most experts agree that an Uplift of more than 10% is very rare.
2. Thrilling zone
With between 10,000 and 100,000 visitors a month, AB testing can be a real challenge. Even if the page you are testing receives half of your total traffic, it would still require an uplift of around 9% to produce reliable results.
3. Exciting zone
With between 100,000 and 1,000,000 visitors a month, AB testing becomes much easier. With this kind of sample size, it is necessary to find an Uplift of between 2% and 9%, to find a winner. However, large volumes introduce a new consideration; you need to make absolutely sure that your conversion rate is maintained whilst you test. With such high traffic, losing even a fraction of the sales you would usually expect would equate to a significant amount of money.
4. Safe zone
Beyond one million visitors a month, you’re in the “Safe” zone. You can run experiments continuously with an iterative, rather than innovative, approach to optimisation.
For a full guide on how to calculate your A/B testing sample size, see: AB Testing Sample Size
Use this simple significance calculator to see if your A/B test results are significant.
In January this year, a Customer Service Software provider Increased their CTR by 300% on a key page in just one month of testing.
By adjusting the text in their CTA button, our client achieved a huge uplift in their KPI.
Test: CTA button text
KPI: Click-through rate
What Are The Most Common A/B Testing Mistakes?
A/B testing can be tricky, and it’s all too easy to invest time and money without producing any results. These are some of the most common mistakes people make.
1. Prioritising the wrong things
You can’t test everything, so you should prioritise your tests based on the following criteria…
- Potential: the potential gain from a successful test
- Importance: the volume of traffic on the tested page
- Resources: What is needed to run the test and apply the results
- Time to market: the delay between launching the test and implementing the changes.
2. Testing Too Many Things
An A/B test involves comparing A with B. In other words, testing one thing at a time. Running too many tests without structuring the process will extend the time each test takes to produce reliable results. Again, this comes down to the question of sample size.
3. Stopping The Test Too Early
In this case, “too early” simply means that you stop your test before the results are both significant and representative. It is one of the most common A/B testing mistakes, and possibly the most important to avoid. Some A/B testing experts advocate a strict “No-Peeking” rule when running a test. That way, they are not tempted to end a test before it has reached the target sample size.
For a list of the most common A/B testing mistakes, see: A/B Testing Mistakes
A winter sports retailer increased their conversion rate from mobile traffic on a category page. The test variation included an on-page notification announcing a bundled offer.
Test: On-Page Notification
KPI: Conversion Rate
Frequently Asked Questions
in A/B Testing
Does A/B Testing affect SEO?
Google clarified its position regarding A/B Testing in an article published on its blog. The important points to remember are:
- Use “Canonical Content” Tags. Search engines find it difficult to rank content when it appears in two places (“duplicate content”). As a result, web crawlers penalise duplicate content and reduce its SERP ranking. When two URLs displaying alternate versions of a page are live (during A/B tests, for example) it is important to specify which of them should be ranked. This is done by attaching a rel=canonical tag to the alternative (“B”) version of your page, directing web crawlers to your preferred version.
- Do Not Use Cloaking. In order to avoid penalties for duplicate content, some early A/B testers resorted to blocking Google’s site crawlers on one version of a page. However, this technique can lead to SEO penalties. Showing one version of content to humans and another to Google’s site indexers is against Google’s rules. It is important not to exclude Googlebot (by editing a page’s robots.txt file) whilst conducting A/B tests.
- Use 302 redirects. Redirecting traffic is central to A/B testing. However, a 301 redirect can trick Google into thinking that an “A” page is old content. In order to avoid this, traffic should be redirected using a 302 link (which indicates a temporary redirect).
The good news is that, by following these guidelines, you can make sure your tests have no negative SEO impact on your site. The better news is that If you are using A/B testing software, these steps will be followed automatically.
Does A/B Testing affect website loading speed?
A/B testing software can reduce loading speed due to the way in which it hosts competing versions of a page. A testing tool can create scenarios in two ways:
- From the client’s side (front-end)
- Using server-side scripts.
Server-side – This form of A/B testing is faster and more secure. However, it is also expensive and more complicated to implement.
Most A/B testing software operates on a client-side basis. This is to make editing a site as easy as possible. In order to reduce the impact of testing on a page, the best A/B testing solutions have found ways to speed up page loading.
How Do You Choose What To Test And When?
You can calculate the number of experiments you can run in a single year by entering the following details into a sample size calculator and finding out the number of days required for each test.
- Your page’s weekly traffic
- Your page’s current conversion rate
- The uplift you can reasonably expect to achieve
Once you know how many tests you can run on which pages, you can put together a full testing strategy for your website. You should start by focusing on the pages that will take the shortest amount of time to produce significant results, as these will tend to be the ones that provide you with the most “low hanging fruit”.
What to test
This will depend on where you are in the A/B testing process. Optimising a website that has a very low conversion rate allows you to take more risks with your edits. You may even decide to try different kinds of offer or value propositions.
For later-stage tests, when you are optimising for more marginal gains, each test should focus on a more discrete variation. These are the kind of experiments favoured by large companies like Amazon and Google.
A/B testing is full of specific technical terms, but most of them have simple definitions. Here are some of the terms we get asked about most frequently.
What is A/B/N Testing?
In an A/B test, your visitor sees version A or version B of a particular page. A test with three page versions might be called an A/B/C test. “A/B/n” is used as shorthand for a test that has “n” different versions.
What is A/B Testing Flicker?
To the flicker effect, you need to ensure your page loading speed is as fast as possible. Convertize does this with its Lightening Mode feature.
Your Control is the unedited version of your webpage. Rather than running an alternative version against historical data, it is important to run both pages simultaneously (even though this will extend the duration of your tests). The reason for this is that your conversion rate can change independently of the design of a particular page. For example, it would be misleading to compare the conversion rate of your Control in November with the conversion rate of your Treatment in December.
A “KPI” (Key Performance Indicator) is a metric used to judge the performance of your webpage. It is important that the KPI you choose is quantifiable and relevant to your business goals. For eCommerce, the most common KPI to choose is a sale. For other industries, it might be a form completion, an email sign-up or any other measurable action.
What is a Multi-Armed Bandit?
The “Multi-Armed Bandit” is an algorithm that directs traffic to better-performing pages. Some A/B testing tools allocate traffic evenly between variation A and B. This could mean directing half of a website’s traffic to a bad version (which can cost large companies vast amounts of money). A Multi-Armed Bandit stops that happening.
What is Multivariate Testing?
“Multivariate” or “Multivariable” (MVT) tests are simply ones in which you carry out A/B tests (or A/B/n ones) on several independent page elements simultaneously. There are different ways of carrying out multivariate (MVT) tests. You can show every single possible combination of page elements (“full factorial”) or only a fraction of them. The various kinds of “fractional factorial” test often have complicated-sounding names (for example, “Taguchi”). You can also choose to display losing combinations less frequently.
What is Server-Side vs Client-Side A/B Testing?
“Server-side” A/B testing tools run on the page’s server. The page is compiled by the server, and is presented to the browser in its finished form. In the same way that some PC software is only available for Windows or for Macs, some server-side software is only available for particular servers (such as PHP).
What is Split Testing?
Split testing is exactly the same as A/B testing, except you create separate URLs for each version of the page.
Your Treatment is the edited version of your page. It is the B version in a classic A/B test.