What is it that defines A/B tests that makes us have to give them their deserved share of importance?

The apparent simplicity of the A/B Test concept, the ease and speed of implementation and the surprising results of the most well-known success stories can make us execute them many times without stopping to think about the decisions and conditions that will define the course of the tests.

And this can lead to a failure of results and frustration to continue using this tool.

However, if we know in detail the scope and capacity of the A/B Tests, the decisions we must make during the preparatory phase prior to their execution and the relevant considerations for analyzing their results, we will increase their chances of success.

Without forgetting that negative results can also be used to draw lessons for the future.

In this article, we will review the most important concepts of the A/B Tests in a context that facilitates their comprehension and scope, and we will also look at what we must take into account in order to carry them out successfully.

Finally, we will discuss a case study on how to implement them with Google Analytics in a content manager as popular as WordPress.

Defining the problem: How to increase conversions?

Imagine that you have a Landing Page with an average of 10,000 monthly visits and a conversion rate of 1%, that is to say, 100 users complete an action that marks a goal in your project.

Without going into assessing whether these are good or bad values (which would be the subject of debate for another forum), how could you double the number of conversions?

In other words, we focus on an intermediate step in the sales funnel:

Conversions as part of the sales funnel

The first, perhaps obvious, solution we usually think of is to increase the number of visits: the more visits, the more conversions’ seems a reasonable statement.

However, this solution is based on the following two assumptions:

  1. That the conversion rate is unchanged. Since we do not change the Landing Page, we tend to think that the conversion rate will not change significantly either.
  2. That the proportion of qualified visitors, within the increase in visits we have received, will maintain a similar proportion to what we have had before.


The harsh reality is that we have no way of ensuring that these factors remain more or less constant.

In other words, even if we manage to double the number of visits, it is likely that the increase in the percentage of qualified visits and the impact on the conversion rate will be lower, so that in the end we will not achieve the desired effect of doubling the total number of conversions.

Even assuming that we manage to keep the conversion rate and the proportion of qualified visitors constant, in order to double the number of monthly visits we will have to develop and launch promotion and dissemination campaigns for our website.

However, implementing these campaigns has costs associated with how we do them:

  • Organic positioning campaigns (SEO), which require continuous monitoring and dedication over time and whose results may take several months to become apparent.
  • SEM (AdWords) campaigns, which require an economic investment during the duration of the campaign and although their results can already be observed from the beginning of the campaign, they only persist as long as this investment is maintained.

Duplicating visits does not guarantee double conversions and, in addition, may require a significant investment, either in time or money.

Let us now look at the other leg of the equation and ask ourselves the following question: what if, instead of increasing the number of visits, we try to increase the conversion rate?

Identifying what can increase the conversion rate

In the scenario we had imagined at the beginning, with 10,000 hits and a 1% conversion rate, we looked at the possibility of doubling the number of hits to double the total conversions, but we saw that we had some drawbacks.

We have another possibility to double the number of conversions if, instead of the number of visits, we focus on doubling the conversion rate.

As soon as you search the Internet, you’ll find hundreds of tips and tricks for designing Landing Pages that increase the conversion rate.

However, can we be sure that these techniques are really effective for our website and how much benefit they can bring us?

I am not questioning the effectiveness of these techniques, nothing further from my mind, but we must not forget that there are no universal solutions’ and that, within the general guidelines that these recommendations set out, there are also multiple factors that affect the extent to which they are effective.

We can take as a reference the market niche in which we are located or the profile of our target audience, to name a few.

As a first approximation to address our problem, we could think about modifying our Landing Page by introducing some of these techniques or recommendations and wait to see what happens.

However, this option has several drawbacks:

  • Most of the time, these recommendations are not specific in their description.

Rather, they refer to general elements whose effect on the conversion rate should be evaluated and assessed for the specific characteristics of our website.

For example, things like choosing one or the other color for the buttons, the typeface of the action call, or the images used may influence the conversions, but what color or font would work best for “my” Landing Page?

  • If we have made a mistake in the application or selection of a technique, we may lose a significant number of conversions.

And, consequently, have negative effects on our profits.

Bearing in mind that observing the effect of a change may require a high number of visits, the cost of lost profits may be too high for our objectives.

  • Results may be influenced by external or seasonal factors.

A very clear example would be if our products or services are related to leisure activities that, most likely, have a greater commercial pull at dates close to holiday periods.

If we change our Landing Page on those dates, we have no way of knowing whether the increase in conversions is due to those changes or to the proximity of the holidays.

At this crossroads, what could we do to be able, on the one hand, to “experiment” with various alternatives to improve the conversion rate but, at the same time, reduce the damage caused by inappropriate changes in the Landing Page or the effect of external influences?

Fortunately, we have a tool, easy to implement and execute, that can help us answer this question: A/B tests.

A/B Tests help to identify which changes in a web page can improve the value of a metric-target

So far we have set ourselves the goal of increasing the conversion rate as an example to better understand the concept of A/B testing, as it is a generally well understood and unambiguous metric.

However, we can also use A/B tests to analyze any other metrics, such as bounce rate or page dwell time.

In short, all the metrics that you want to optimize and that you can analyze from very specific modifications within a page.

What are A/B Tests?

A priori, we cannot know for sure what changes to a Landing Page would lead to an increase in conversions, so we need to experiment to find the best combination, but reducing the risk of losing profits or obtaining inconclusive results due to external influences.

In this sense, A/B tests allow us to compare the behavior of two versions of the same page, usually a Landing Page, that differ in a single element and whose objective is to evaluate the impact that this element has on the visiting users, measured by the metric that we define, during a certain period of time.

During the test run, visitors randomly view one or another version of the page, so that at the end of the experiment, the total number of visitors will be divided equally between the two versions.

It could be seen as if, when a user requests the page, a coin is flipped over and, as heads or tails come out, one version or the other is shown.

With this method, we solve two of the problems we identified earlier, when we considered modifying the Landing Page without any further consideration:

  1. Since the two versions of the page coexist, we eliminate the influence of any external or seasonal factors on the analysis of the results, since both are affected equally.
  2. If the change adversely affects the conversion rate, the impact on the loss of our profits is reduced by half.

The classic example in the A/B tests refers to the color of the buttons in the calls to action, using one color on the original page (“A” version or control page) and another color in the variation (“B” version) of the page:

One-page A and B versions with different colored buttons

However, we should not limit these tests to changes as simple as variations of colors, fonts or even headlines and images, but also structural changes in the layout of different elements of the page, such as the length of an article or the position of an advertising banner:

A and B versions of the page with a banner in different positions

Don’t be afraid to try any changes you can think of, but always be careful to avoid multiple or radical changes that make the two pages too different.

If we do so, we will not be able to identify which change has produced the greatest benefit or even the positive effects of one change can be neutralised by the negative effects of another and we will be discarding any change that would have been beneficial.

Whatever you change, never forget the maximum of the A/B tests: two similar versions of a page, with a single different element between them.

To give you an idea of the importance of being creative with the changes we can make, take a look at the following two announcements, which one do you think has the highest conversion rate?

Two one-page versions with a change in the main picture

The information on both pages is identical but, while on one page, the baby looked at us head-on, on the other page, he looked at the headline of the ad… the latter being the one with the highest conversion rate!

Therefore, do not limit your imagination as to what kind of changes you want to experience.