A/B testing

The point of an A/B test is to separate the effect of your change from everything else that varies, such as seasons, campaigns and luck. Because both versions run side by side to comparable visitors, a clear difference in results can be put down to the change itself.

It is not the same as changing a page and comparing this month with last month. That before-and-after comparison mixes your change with every other difference between the two months, which is why it often misleads. A sudden rise after a change may simply reflect a holiday, a news story or a campaign that happened to start the same week.

How it works

A testing tool or your own code randomly assigns each visitor to version A (the current one) or version B (the change) and keeps them in that group on return visits. Both groups are measured on the same goal, recorded through conversion tracking. When enough people have taken part, a statistical test estimates how likely it is that the difference is real rather than chance.

How many people "enough" means depends on your current conversion rate and the size of the change you hope to detect. Small improvements need far more visitors to confirm than large ones, which is why sample-size calculators exist and why tests should run for whole weeks to include weekday and weekend behaviour.

There are edge cases to plan for. Returning visitors who clear cookies or switch devices may see both versions, which slightly blurs the result. Seasonal events, a sale or a newsletter blast during the test can change who arrives, although random assignment means both versions feel the same effect. A result that is "statistically significant" at the usual 95% confidence level still has roughly a one-in-twenty chance of being a false alarm when there is no real difference, so if you run many tests, expect some winners to be flukes. Large, important changes are worth re-checking after launch by watching whether the gain holds over the following months.

Example

An online flower shop tests two versions of its product page. Version A lists delivery cost only at checkout; version B states it under the price. The test runs for four full weeks with visitors split evenly. Version B produces more completed orders, and the difference holds up statistically, so the shop keeps it and tests the next idea.

Before starting, the shop calculated that it needed roughly two thousand visitors per version to detect the size of change it cared about, and its traffic made four weeks realistic. It decided in advance that completed orders, not clicks on the basket button, would decide the winner, and it did not look at the result daily, which removed the temptation to stop early when version B led after the first weekend.

Why it matters for your business

A/B testing protects you from expensive opinions: it shows whether a redesign or new headline actually brings more business before you commit to it everywhere. The cost is time and traffic. A site with a few hundred visitors a month may wait a very long time for a clear answer, and in that case careful research and sensible changes are a better use of effort than a test that never finishes.

Common mistakes

• Stopping the test the moment one version pulls ahead, which greatly raises the chance of a false winner.
• Testing tiny details on a low-traffic site where no result could ever become reliable.
• Changing several elements in version B and then claiming one of them caused the difference.
• Letting the testing script slow the page or make content flicker, which distorts the result and hurts Core Web Vitals.

Questions people ask

How long should an A/B test run?

Run it until you reach the sample size you calculated beforehand, and for at least one or two full weeks so every day of the week is represented. Ending earlier because the numbers look exciting is the most common way tests produce wrong answers.

What is the difference between A/B and multivariate testing?

An A/B test compares whole versions, usually two. A multivariate test changes several elements at once in different combinations to see how each contributes. Multivariate testing needs much more traffic, so most small and mid-sized sites are better served by simple A/B tests.

Can A/B testing hurt my SEO?

Not when it is done properly. Google's guidance is to avoid showing search engines different content from users, use temporary redirects rather than permanent ones for split URLs, and end tests once they are finished. Short, honest tests are a normal part of running a website.