Reading Time: 14 minutes
Page Speed: How to Avoid Loading Speed Issues in A/B Testing
4.8 (96%) 65 votes

The challenge of any SaaS provider is to continuously find ways to improve its software. As Convertize has evolved, we have worked hard to develop our A/B Testing system so that it is secure, reliable and easy to use. One of our priorities has been to ensure we deliver the best page load speed possible whilst running an A/B test.

Page Speed &  A/B Testing

Site loading speed affects search engine rankings and conversion rates. A one-second delay in page loading typically results in 7% fewer conversions and 11% fewer page views.

Browsers can identify images in as little as 0.13 milliseconds and the Page Flicker associated with slow loading times can seriously affect user experience. It can also lead to lower search engine rankings.

Our solution has been to create a small piece of script (called a “pixel”) that users can install on their website in a single straight-forward process. Not only does this make our system easy to use, but it allows for speedy load times. With “Lightning Mode,” Convertize achieves page loading speeds 17.2 times faster than other leading software.

Our pixel is now installed on thousands of sites. With over half a million requests per day, we have been presented with two challenges:

  1. How can we ensure our pixel is served as fast as possible
  2. How can we ensure our pixel runs an experiment as fast as possible

This article will give an overview of the technologies and concepts used in order to answer these questions. We conclude with the AB Testing benchmark we established, showing that our pixel provides now up to 17.2 times faster AB Testing than a well-known alternative platform.

The process begins in our SmartEditor™, a suite of editing tools built using AngularJS on the front-end and Symfony on the backend, handling API requests.

As important as it is to make our pixel operate fast on our customers’ websites, we wanted to make sure the user experience of the Convertize application was also a pleasant one. We have therefore made a number of optimizations to ensure this is the case.

The first step is to load the webpage into the application for the editing suite to be enabled. This happens using the Convertize SmartProxy™, which will retrieve the page and all assets using cUrl in order to have direct control over them for the purpose of editing.

The SmartProxy™, written in PHP 7, has been designed to override native Javascript functions in the browser to ensure the webpage behaves inside the application as if it were accessed directly.

Each modification introduced by the user is called an Action. The user is able to introduce many changes to the webpage using the suite of tools provided.

Every time an action is made such as inserting an image or rearranging a bullet list, this is sent to the backend to be processed via an ajax request. To ensure we send the smallest amount of data to be processed, only the action being applied will be sent to the backend processor.

Decorators describe DOM elements which exist in the original DOM tree. A single decorator represents one unique element (based on XPath). It contains information about the final version of the element.

If the user will introduce different changes on the same element, such as changing CSS color and then modifying the text, all modifications are merged into one representation of the element called a Decorator. This has the benefit that the pixel can quickly apply a set of changes on a single element and there is less data to store in the pixel, resulting in a smaller file to load.

Let’s assume that a user introduces the actions shown below (represented with JSON data). The example is simplified by introducing modifications on the same element, but the idea is the same as introducing modifications on multiple elements in a different order.

lightning mode

Figure 1. Example list of modifications of single element

All these changes will be merged into one single “decorator” action which represents the final version of the specified element. This object will be used inside the pixel and during initialization of the SmartEditor™.

Figure 2. Example decorator based on Figure 1

For each element that has been modified in the editor, the pixel will apply that modification to the live webpage. It does this with O(1) complexity, meaning that if the same element is modified multiple times in history, the pixel will only store the very last modification.

Let’s assume that you have three paragraphs on your website and you introduce 30 text modifications for each of them (90 modifications in total). Rather than store all 90 modifications in the pixel, it will store only 3 (one modification per paragraph). If the paragraphs contain 500 characters, the pixel will grow by only 1500 bytes.

DOM actions represent modifications which affect the structure of the Document Object Model. As an example, it can be rearranging a list, html editing and inserting an image merged into one unit called “DOM”. The key thing is that the backend merges all actions for the same element into one representation. See the following example JSON showing a number of editing actions: an inserted element, a couple of text changes followed by a css attribute update.

lightning mode

Figure 3. Example modifications based on insert action

All these changes will be merged into one action to be applied by the pixel, as follows:

Figure 4. Merge DOM modifications

It is important that the system is highly scalable. This not only means handling more traffic and load, but the system should not suffer performance issues when the amount of data increases.

When it comes to making modifications in the SmartEditor™, this means that if you edit the experiment and make 10,000 edits to it, the last edit will be processed just as fast as the first one .

How do we accomplish this? We create a “build” which contains the minimal amount of actions that the pixel must process. When a new edit is made, the optimization process will merge this into the existing build to generate a new updated build of data for the pixel. This process ensures data can be updated incrementally.

Nowadays websites are rarely static and there are a number of Javascript libraries being used to provide dynamic functionality such as newsletter opt-in modals, shopping carts and carousel of images.

The SmartEditor™ has been designed to allow you to edit dynamic elements. It does this by listening out for DOM Mutations, i.e. when a 3rd party Javascript library will modify the DOM structure of the page.

After all modifications are applied, the system registers a Mutation Observer. When a new DOM node is added system attempts to match non-applied changes on this element and its children, which will allow users to apply the editing tools to the dynamic element.

Advanced users who have worked with the HTML markup language are able to modify the HTML directly. The pixel is smart enough to only store the data for the elements that have actually been modified rather than text for the whole element.

This has the advantage of not storing unnecessary data, resulting in a smaller and faster pixel. No unnecessary changes are applied and therefore the page load time is decreased.

Let’s assume that a user has introduced the changes presented below. On the left side we have the original DOM structure of the specified element and on the right side we have the modified structure.

Figure 5. Example of advanced HTML modifications


The user has introduced many fundamental modifications. Both the HTML before editing and after editing will be compared using the diffDOM library. Let’s see an example of this using the online Pretty Diff tool.


Figure 6. Output of Pretty Diff

After a diff has been performed, only the elements that were changed will be added to the list of actions to process and add to our “build”.

When the Advanced HTML editing feature is used, the application will make sure the HTML being entered is valid and not allow modifications to be saved if they break validity. You will see a red X where the problem occurs.

Figure 7. Example of HTML validator

The Convertize pixel is a Javascript file containing the core code to modify a webpage along with the optimized data saved from the SmartEditor™. Our engineers and developers found 7 key areas to optimize, ensuring that our Pixel does not affect page loading speed.

  1. Javascript
  2. Server Architecture
  3. HTTP File
  4. Pixel Shrinking
  5. Responsive Page Hiding
  6. Asynchronous Tracking
  7. Conversion Cookies

The pixel works with core Javascript instead of 3rd party libraries like JQuery to keep the file small and to ensure it will work well in all browsers/devices supporting Javascript.

As our customer base grew we knew it was time to re-architect our server infrastructure. We migrated from a Google Cloud environment to Amazon Web Services, at the same time introducing docker containers and autoscaling.

The pixel is served by a set of containers managed by an Application Load Balancer (ALB) in an ECS Cluster. The container itself runs NginX to serve the Javascript files from EFS storage and Gzip them to reduce the size further.

We will write about our move to AWS in another article and how it now allows us to handle any amount of traffic by scaling our services up automatically when required. Here is a diagram showing an overview of the AWS setup in regards to serving the pixel.

Figure 8. Server architecture of Pixel setup

A webpage contains many assets (images, Javascript, CSS) which all must be loaded before the page is ready to be presented to the user. The Convertize pixel loads as fast as possible by ensuring:

  • All data, code and css required are embedded into a single file. Browsers can only handle so many concurrent connections per domain so the less HTTP connections the better.
  • Any user detection information such as Geolocation and Device Detection are injected directly into the pixel by NginX so we don’t have to make further 3rd party API calls.

A user can enable or disable certain features for their experiment. For example, they may choose to modify the page only, or also add a SmartPlugin™.

To keep the pixel download size as small as possible we decided to take into account what is actually trying to be done in the experiment and only download the code necessary.

For example, as soon as the user turns off all experiments, the pixel becomes 0-byte in size. If they don’t use any SmartPlugins then the code for those plugins will not be part of the downloaded file.

In order to avoid the FOOC issue (Flash of Original Content, also called Flickering effect), the pixel will briefly hide the content of the page until all SmartEditor™ modifications have been made.

Is this always necessary? What if the only change being made was to add a SmartPlugin™ like Scarcity onto the page? In this instance, there is no need to hide the original content because it is not being modified, we are only displaying a notification on top of the existing page.

The pixel is smart enough to detect the type of changes required for the experiment and perform as optimally as possible, ensuring the user will see the page appearing as soon as possible. Following is an example flow the pixel will follow to apply a split url variation.


Figure 9. Pixel flow for Split URL variation

When an experiment is loaded onto a page, one of the tasks of the pixel is to record goals. This is accomplished by sending a HTTP request for a blank image with parameters to the tracking server and then listening for a successful status code.

It is important that these requests are non-blocking and aysnchronous where possible, to ensure the page load speed isn’t afffected and that the tracking happens behind the scenes without the user being aware. Following are examples of optimizations made by the pixel in relation to goal tracking:

  • Page View goals are handled only when the experiment has been loaded into the page and the page has been displayed to the user. This is because the user experience has a much higher priority.
  • When sending these tracking requests, we don’t need to wait for a response unless the variation being applied is a Split URL variation, in which case we must make sure all tracking was recorded successfully before redirecting the user.

This is an area that has improved greatly to ensure backend processing of conversion data happens as efficiently as possible considering the high number of requests being sent per day.

With our previous setup we had an NginX server listening to conversion requests and logging the data into a file which was read by a Treasure Agent Daemon. The daemon would parse the log data for conversion and experiment participation entries and then spawn a script to insert the conversion into KairosDB (a timeline series interface to Apache Cassandra).

NginX was configured with the ngx_http_userid module to create 3rd party cookies automatically in order to identify clients by assigning them a unique identifier. The conversion script would query KairosDB with this unique identifier and determine whether the conversion was new or not (this is especially important to identify new vs returning visitors).

Following is a diagram showing our previous setup.

Figure 10. Old architecture for processing conversions

What we learnt as traffic grew was that this setup could not scale. Here were some of the problems we faced:

  • Storing and querying KairosDB for unique identifiers was not a scalable solution. The more conversions we had, the slower the system became.
  • Using Treasure Agent in this way was limited by the speed at which it could parse log files. We were unable to scale this out as the log entries had to be read in sequential order to handle cross-domain tracking.

After our engineers thought about what we needed to accomplish, we re-architected a solution that would automatically scale in our AWS infrastructure and apply conversion processing 200 times faster.

Our new solution has said goodbye to Treasure Agent and the NginX module for visitor identification. When the pixel will send a tracking request to the server, it is retrieved by NginX running inside a scalable docker service, which will proxy this through to a python WSGI application.

The application will now store all experiment participation and previous conversion data into a 3rd party cookie, meaning that identification of new vs returning users and conversions is done instantly without the need to query a database.

If the conversion is one we wish to track, it will be sent to an Amazon SQS queue for it to be handled by consumer python scripts that will pop items off the queue and process them, inserting data into KairosDB. As there is no querying to perform anymore for uniqueness, we simply insert data and write operations in KairosDB are lightning fast. We improved this speed further by creating a KairosDB and Cassandra cluster behind load balancers.

Here is our new setup showing the interaction between the browser making the conversion request and it finally getting tracked.


Figure 11. New architecture for processing conversions


We have conducted a loading and speed benchmark against our competitors (for propriety sake we have omitted their name and use XXO throughout instead).

We first provide a benchmark against pixel size including just core functionality. The size affects how much data must be downloaded before your A/B test code can begin to run. Following on from this, we will make some different A/B tests, the typical modifications a user would make on their site, and compare 3 results:

Pixel Size – How much data is being downloaded
Pixel Load Time – How much time it takes to download and run the pixel code
Page Load Time – How long before your A/B test has been applied and the variation is displayed

Basic A/B testing benchmark information

  • Internet connection speed: 20Mbps
  • Website size: ~1223KB
  • DOM tree size: 732 elements
  • Page loading time has been tested using GTMetrix tool
  • Original page load time (without any pixels): 4.40s
  • GZip compression disabled

Loading all pixel code with no modifications

Our Pixel has a few key advantages compared to the competition:

  • It doesn’t load any additional resources (JS/CSS) so its size is very small (56kB for all code). It requires only one HTTP request to our servers. XXO sends at least 4 different requests which has bad impact on the loading time.
  • It virtually doesn’t slow down customer’s website. Loading time is very close to the original loading time
  • It’s easy to install – one simple tag vs piece of JS code (XXO)

Comparison benchmark

Test #1

Benchmark steps:

  • Modify text of a 676 bytes long paragraph
  • Insert new paragraph with 128 bytes of text
  • Change content of the inserted paragraph (extend content with additional 600 bytes of data)
  • Advanced HTML action on the 4645 bytes long container
    • Add attribute to the element A (data-attribute=”test”)
    • Change style for element B (add color and background color)
    • Remove DOM element (40 bytes long)
    • Rearrange elements
  • Insert new image

Test #2

Benchmark steps:

  • Modify text of element A – remove 100 bytes
  • Modify text of element A – add 50 bytes
  • Modify text of element A – replace content with new text (128 bytes)
  • Advanced HTML on element A – add H1 (30 bytes)
  • Remove element A
  • Undo changes
  • Modify text of element A – add 1024 bytes

Test #3

Benchmark steps:

  • Rearrange element A after element B
  • Rearrange element A after C
  • Rearrange element A after D
  • Rearrange element A before element B (original position)
  • Advanced HTML on the parent of element A
    • Rearrange element A after element B
    • Add attribute to element A
  • Change style of element C
    • Change color
    • Change background-color
    • Add border
  • Use advanced CSS editor for element C
    • Remove all properties (should be original state)

Test #4

Benchmark steps:

  • Modify text of element A
  • Modify text of element A
  • Modify text of element A
  • Revert original text of element A (Pixel should do nothing)

Loading Time Results

Pixel size

Pixel load time

Page load time


You can see out of these tests that our pixel is now up to 17.2x faster than XXO, which is massive difference!

About the future

In this article, we have outlined SmartEditor™ and Pixel optimisations–the result of our hard work, and our dedication to providing you with a better experience. This development signifies enormous improvement and innovation to our optimisation platform, allowing our solution to work smarter and faster than any other optimisation product on the market. And yet, we are aware that this is still just a stepping stone along the way to perfect optimisation. When you’re committed to growth as much as we are at Convertize, “progress is inexhaustible” and change never stops. We will continue to look for ways to make your experiences smarter and faster.

With that in mind, enjoy Lightning mode and stay tuned!

Author note: I would like to thank David and John for their hard work and endless knowledge. Without their extensive contribution none of this would have been possible.


by Benjamin Ligier

Benjamin is a CRO Expert at Convertize. He is passionate about design, web marketing and consumer psychology.

Newsletter signup

Get your FREE copy