What is an A/B test?

The A/B Test is a methodology for comparing two versions of a webpage, app, audiences or ads to determine from a comparison variable, which one brings the best result.

What is your goal?

The A/B Test has the main objective of generating a performance increase, such as:

  1. Raising the Conversion Rate;

  2. Reducing the website's Bounce Rate;

  3. Expanding the expectation of sales/lead among audiences.

It is possible to separate the tests by marketing funnel stage, reaching audiences at different moments in their conversion journey. Check the image below; different factors can be tested, such as ads, Landing Pages and welcome emails:

What can I test?

Let's separate these A/B Test options according to the marketing funnel stage:

  • Ads – It is possible to test ad variations, such as changing the copywright, making changes to the creative (in the case of an image), replacing the color of a CTA or even the button phrase!

  • Landing Page – The universe of LP’s is even broader. You can change the main banner, increase the number of Calls-to-action (CTAs) on the page, change colors, replace fields on a form, show / hide the service price of a product ... Among other variations.

  • E-mail – E-mails have similar aspects to the ads, but in this case we already have the customer's e-mail. Thus, it is assumed that there was a first contact with your company (so we are in a bottom of funnel stage). Therefore, it is possible to change email communication, test the use of category or product images, CTA’s, redirect links and more variations.

How does testing impact performance?

For a B2B company, the main focus is to increase the volume of qualified leads.

In terms of e-commerce, the ultimate goal is to increase sales volume and generate greater revenue.

However, in both cases, the A/B test can solve the following points:

a.  Improve the user experience;
b.  Improve the traffic quality;
c. Reduce the bounce rate;
d. Make controlled changes to the site with low risk;
e. Have results with statistical reliability.

How does an A/B test work?

It is possible to separate the A/B test into five stages: Research, Observation and Hypothesis Formulation, Creation of Variation, Execution and Result Analysis.


Research consists of analyzing the current data. It seeks to answer the main question: "How am I today?".

It may seem simple, but for each test, we have a data platform, with different metrics of analysis. Creating an A/B test without proper research can cause minimal performance impacts.

Therefore, applying the Pareto principle (80/20) it is important to align the test strategy with the objective that wants to be achieved.

For each test we have a specific targeting platform, including:

  • Google Analytics - Test on the site as a whole;

  • Facebook and Instagram - Audience and ads test;

  • Google Ads - Audience and ads test;

  • Firebase - App testing

Observation and Hypothesis Formulation

This is the most critical step! Creating a test without a hypothesis is analogous to taking a sea voyage without a compass. In other words, it is to follow a path without direction and with the uncertainty of getting somewhere.

Before we continue, let's go deeper into the concept of hypothesis. The hypothesis is a statement created in simple language that must be challenged.

Let's look at an example of Hypothesis applied to a Pareto ad:

Hypothesis: Ads from the Pareto Brand campaign have a higher number of clicks when using the term "Free Trial" in Headline 2.

Variation Creation

Based on the hypothesis, we will put the statement to the test, creating a variation group and a control group (the original).

It is possible to create a test with multiple variables, however, the more simple the test, the shorter its duration. And the greater the effectiveness in determining the real reason for the performance improvement! See this other article for an explanation of Statistical Ad Experiments.

Continuing the example of Pareto:

Control Group:

Experiment Group:


Having created the variation, we have to put the test into practice!

It is important to remove any biased variables applied during the test. For example, when testing the ad above, it’s ideal to ensure that the number of impressions or investment for each variation remains as close as possible.

Result analysis

After the AB Test is finished, it is time to analyze the main results generated. Important performance metrics such as: Cost, clicks, CTR, conversions, CPA or ROAS, among others, should be evaluated.

Let's check the example:

Control Ad

Cost: R$4.000
Clicks: 3.785
Click-Through Rate (CTR): 4,25%
Conversions: 112
Cost per action (CPA): R$ 35,12

Variation Ad

Cost: R$4.050
Clicks: 4.003
CTR: 5,12%
Conversionss: 96
CPA: R$ 42,18

What is the best ad?

Following the hypothesis tested above, that the term “Free Trial” would increase the number of clicks, we have that the winning ad is Variation. Both the number of clicks (+ 5.7%) and the CTR (+ 20.5%) are higher when we have “Free Trial” in the Headline 2 of the Ad.

However, analyzing it in terms of conversions, it is possible to notice that the number of conversions fell from one variation to another, thus having a 34.6% increase in CPA.

Click-wise, in fact the example shows a superior performance!

Conversion-wise, however... the answer is the exact opposite!

Now just apply the test on your ads!

And what is the analysis with Pareto Ads?

On Pareto Ads, we have a specific One Click for A/B tests! See the example below:

It identifies the test being applied to your Google Ads account, evaluating the account's ads.

In the same One Click, you can see which test was done! In this case, the main variation is the change of Headline 2.

Then, it presents the test's metrics, with the statistical result and already presents the winning and losing ad! In this case, the measurement metric is the CPMI (Conversions per Thousand Impressions).

And most importantly, it runs the test with just one click on the “Pause” button, to pause the losing ad!

See an example with the One Click screen for “Ad Experiment”, with all the information of the ads and the button to execute it:


In this article, you learned more about what an A/B Test is and how Pareto Ads helps you apply it in your ads account.

After completing an A/B Test, we can proceed with the test schedule, with the aim of further improving the quality of the ad. New information can be tested successively. This virtuous cycle is fundamental for the optimization of campaigns.

Therefore, an A/B Test is an excellent opportunity to save money and make your digital marketing campaigns even more effective.

Like it? Visit our blog!

Did this answer your question?