Reading Time: 5 minutes
Where To Start With A/B Tests
4.5 (90.23%) 43 votes

Getting started is always the hardest part, and A/B tests can be overwhelming. There are countless guides to A/B testing, but they often take for granted the most obvious questions: What should I test? Will it be worth it? Does A/B testing really work? This article answers those questions, and will help you to focus on the things that really matter to your website.

Before you start testing your content, there is one step you can’t afford to skip. Making sure you have a reliable analytics tool will help you to define your goals, choose the best pages to test and prioritise your ideas. Google Analytics is the default option because it gives you detailed insights and integrates easily with Google’s other marketing tools (Google Ads and Search Console, for example). However, you should check your tags are working on a regular basis and pay attention to common errors (like not filtering out internal traffic).

Any A/B test is based on conversions; you’re comparing how many conversions you get from the original webpage and an alternative version.

However, there is an endless list of things you could track as your primary goal: button clicks, video plays, session length, etc… So what should you define as your goal?

It’s important to distinguish between “True Conversions” and “Micro Conversions”. 

  • A True Conversion is any goal that represents a real increase in revenue. A visitor making a purchase or signing up for your app is a True Conversion. 
  • A Micro-Conversion is a goal that may or may not increase your revenue. Whilst it might be good for a visitor to watch a promotional video or look at your pricing page, it doesn’t necessarily mean they will buy anything. 

In general, A/B tests work best when they are based on a True Conversion. That’s because it’s easy to increase the number of people who enter your sales funnel, only to leave when they reach the checkout. A positive A/B test based on a Micro Conversion might not tell you anything about which page version delivers more revenue. 

How Do A/B Testing Tools Track Sales and Sign-Ups?

A/B tests use Cookies to split your traffic into two groups. One group will see the original page and the other will see the alternative. The same Cookie is later used to assign conversions to one of your page versions. For that to work, you need to set a trigger. You can do that in two ways: 

  1. Setting a URL as a goal: by setting a “Thank You” page as a goal, you can ensure that only visitors who make a purchase are counted as conversions. 
  2. Tracking button clicks: by tracking a button as a goal (ideally one that confirms a purchase) you can track True Conversions.

What If My Website Doesn’t Collect Sales or Sign-Ups?

If your website is based on a different kind of customer journey (focusing on driving phone calls or arranging a consultation, for example), you can track different web elements. However, it is important to remember that the uplift you generate represents an increase in leads rather than revenue. 

You can’t test everything, and most websites only have one or two pages with enough traffic to run quick, conclusive tests. So, to prioritise which pages to put the most time and effort into, A/B tests usually involve a priority table similar to the one below. It gives a set of key metrics that help to identify the most important pages and the ones with the best testing potential.

Although the Home Page has the potential to run the largest number of tests, it is only the second highest priority. That’s because the impact of a change made to your Home Page is likely to be smaller than one made to your Basket or Checkout pages. 

As a rule-of-thumb, pages that are more closely connected to your Checkout Page (Landing Pages, Basket Pages and the Checkout itself) tend to have a greater impact than those encountered by only some of your visitors. They also have an added advantage, in that an Uplift further along your funnel is less likely to produce a “Revenge Effect” at a later stage.

To understand how people are using your website, you need to use CRO tools to gather real data. The most important of these is undoubtedly your website analytics. Alongside GA, you could experiment with tools such as on-page surveys, heat-maps, and even session-recordings, to give you a sense of how your users interact with your design.

There is no short-cut to understanding consumer psychology. In fact, one of the most important benefits of running A/B tests is that it allows us to avoid assumptions about how people are experiencing a website.

However, becoming familiar with some basic psychological principles can help to structure your ideas. For example: 

  • Cognitive Load is the total amount of mental effort involved in processing information. It is best to keep this to a minimum (using visual cues, simple language and structured text) because excessive load makes visitors less receptive to new information. A famous study from 2006 found that “processing fluency” (which reduces Cognitive Load) could be used to predict success on the stock market.
  • Implicit Egotism describes the fact that most people make sense of the world around them by relating it to themselves. At the same time, most people gravitate towards people, locations and even objects that share aspects of their identity; they are positively biased towards characteristics that match their own. Because of this, user-orientated information (such as outlining the benefits of a particular product) can have a big impact on behaviour.
  • Framing is the way in which new information takes its meaning from the context in which it is received. The framing of key information (particularly prices) will have a big impact on your conversion rate.

By combining these general principles with real data, you should be able to identify ways to improve your key pages.

Unless you have significant traffic (over 100,000 visitors a month on the relevant pages) multivariate testing is not a plausible option. So, to make sure you know which changes are responsible for any uplift you produce, you’ll have to test one thing at a time. You want to test the ideas that:

  • are most likely to work
  • will have the largest impact
  • will affect the KPI you care about
  • relate to substantial (rather than aesthetic) value

A different kind of priority table can help you to decide which of your ideas to test first.

a/b test ideas

By scoring your ideas on each of these categories, you can give them a number to compare against the others.

One of the most important decisions to make when starting out with A/B tests is which software you are going to use. There are over 20 popular A/B testing tools available, each with different features, capabilities and prices. This set of 10 questions will help you find the best A/B testing tool for your website.

by Benjamin Ligier

Benjamin is a CRO Expert at Convertize. He is passionate about design, web marketing and consumer psychology.