What Is A/B Testing?

A/B testing, also called split testing, is one of the most reliable ways to improve marketing performance because it replaces guesswork with evidence. Rather than debating whether a green button converts better than a blue one, you show each version to a random slice of real visitors and let their behavior settle the argument. Done well, A/B testing turns a website or campaign into a continuous learning engine, where every change is validated before it is rolled out to everyone.

The idea is simple, but the discipline required to do it correctly is where most teams stumble. A test that is stopped too early, measures the wrong thing, or lacks enough data can point you confidently in the wrong direction, which is arguably worse than not testing at all because it dresses a bad decision in the authority of numbers. This guide explains how A/B testing works, the math that keeps it honest, and the mistakes that quietly undermine results. Master those fundamentals and A/B testing becomes one of the highest-return habits a marketing team can build.

How does A/B testing work?

An A/B test starts with two versions of the same asset. Version A is the control, usually your current design. Version B is the variation, containing one deliberate change you want to evaluate. Testing software randomly assigns each incoming visitor to one version and records how each group behaves against a chosen goal, such as clicks, sign-ups, or purchases.

Randomization is the heart of the method. Because visitors are split randomly, the two groups should be statistically similar in every respect except the change you introduced. That means any meaningful difference in outcomes can be reasonably attributed to the change itself rather than to differences in the audiences. Without random assignment, external factors like traffic source or time of day could quietly bias the result. This is also why you should run both versions simultaneously rather than comparing this week’s version B against last week’s version A, since running them at different times reintroduces exactly the outside variables that randomization was meant to cancel out.

What makes a good A/B test hypothesis?

Every solid test begins with a hypothesis, not a hunch. A useful hypothesis names the change, the expected effect, and the reason behind it. For example: Shortening the checkout form from ten fields to five will increase completed purchases because it reduces friction. That structure forces you to be specific about what you are changing and what success looks like.

A clear hypothesis also protects you from a subtle trap: deciding after the fact what the test was about. If you define your primary metric up front, you cannot be tempted to cherry-pick whichever number happened to move. A test with ten possible success metrics will almost always show one that improved by chance, so committing to a single primary metric before launch is essential discipline. This connects directly to the broader practice of conversion rate optimization, where structured testing drives steady, defensible gains rather than one-off lucky wins.

Measuring the results of a test

The core comparison in most A/B tests is conversion rate, the share of visitors in each group who completed the goal. The formula is simple:

Conversion rate = Conversions / Visitors in that group

Suppose you run a test with the illustrative numbers below. These are rounded examples to show the math, not benchmarks.

Version B converted at 6.0% versus 5.0% for A. The relative improvement is (6.0 - 5.0) / 5.0 = 20%, a one-percentage-point absolute lift that represents a 20% relative increase. Both framings are correct, but they sound very different, so always state which one you mean. Reporting a headline that shouts a 20% jump while quietly relying on a one-point absolute gain can mislead stakeholders, so clarity here protects your credibility. Understanding how to improve your conversion rate depends on reading these numbers precisely.

What is statistical significance in A/B testing?

A difference between two versions might be real, or it might be random noise. Statistical significance is the tool that helps you tell them apart. It estimates the probability that the observed difference happened by chance alone. Many teams use a 95% confidence threshold, meaning they accept a result when there is roughly a 5% or smaller chance it was a fluke.

Significance depends heavily on sample size and the size of the effect. Small differences need large samples to prove; large differences can be confirmed with less traffic. This is why low-traffic sites often struggle to run conclusive tests, and why declaring a winner after a handful of conversions is misleading. The result may look dramatic but rest on far too little data to trust. It also helps to remember that statistical significance tells you a difference is probably real, not that it is large enough to matter commercially; a tiny but significant lift may not justify the effort of implementing it.

Planning test duration and sample size

There is no fixed answer for how long a test should run, because the right duration depends on your traffic volume and the effect you are trying to detect. A sensible practice is to calculate the required sample size before you launch, using an online calculator that accounts for your baseline conversion rate and the minimum improvement you care about. Then you run the test until it reaches that sample, and not a moment less.

Duration also matters for capturing natural variation. Buying behavior differs by day of week, so running a test for full weekly cycles avoids skew from, say, a strong Monday or a slow weekend. Ending a test the instant it looks good, a habit known as peeking, is one of the fastest ways to fool yourself with a result that later evaporates. The more often you check a running test and stop at the first favorable moment, the more likely you are to lock in a random high point rather than a genuine effect, which is why committing to a sample size in advance is such an important safeguard.

What are common A/B testing mistakes?

The most frequent errors are predictable once you know them. The table below summarizes several and the discipline that prevents each.

Testing several changes simultaneously deserves special mention. If you alter the headline, the image, and the button color at once and conversions rise, you cannot know which change did the work, or whether one helped while another hurt. Isolating a single variable keeps the result interpretable, which is what separates a true A/B test from a vague redesign. Reliable tracking, often set up through Google Tag Manager, ensures the data feeding your test is trustworthy in the first place, because even a perfect test design produces garbage if the underlying measurement is broken.

When A/B testing is not the right tool

A/B testing excels at optimizing within an existing design, but it is poor at discovering entirely new directions. It answers narrow questions well and broad questions badly. If your traffic is very low, you may never reach significance, and qualitative research, such as user interviews or session recordings, will teach you more per hour invested. There is little point running a test that would take a year to conclude when a handful of customer conversations could surface the same insight in an afternoon.

It is also the wrong tool for changes you must make regardless, like fixing a broken checkout or meeting a legal requirement. In those cases, testing only delays a necessary fix. And for bold, transformational redesigns, a single A/B test can obscure which of many changes mattered, so those are better evaluated over a longer horizon with multiple follow-up tests. A sensible program treats testing as one method among several, reserving it for decisions where the traffic supports a clear read and the outcome genuinely hangs in the balance. Used thoughtfully, A/B testing sharpens decisions where the stakes and the traffic both justify the wait; used indiscriminately, it becomes a bottleneck that slows the very improvements it was meant to accelerate.