Split testing that actually tells you something

Most split tests are decorative. Somebody changes a button from green to orange, watches the numbers for three days, declares a winner, and learns nothing that will ever apply again.

Testing is worth doing. It is just worth doing properly, because a bad test is worse than no test. It gives you false confidence and sends you off rebuilding things that were never the problem.

What a split test really is

You show version A to half your traffic and version B to the other half at the same time, then compare how many people did the thing you care about. Same period, same traffic sources, same offer. The only difference is the change you are measuring.

That last sentence is where most tests fall apart. If version B also went live the same week you started a new ad campaign, you are not measuring your page. You are measuring the campaign.

Test the big things first

There is an order to this, and it is not the order people usually follow.

The offer. What you are actually promising, what it costs, what it includes, what the risk is for the buyer. Nothing on the page matters if the offer is not compelling. This is the biggest lever by a wide margin, and almost nobody tests it because it feels like more work than changing a heading.

The headline and the promise. The first sentence decides whether the rest gets read. Testing two genuinely different angles beats testing two phrasings of the same angle.

The page structure. Where the proof sits, how early the price appears, whether the form is at the top or after the explanation, how long the page is.

The form and checkout. Number of fields, one step or two, what the button says.

The details. Colours, images, microcopy. Real but small. Do these last, when the big questions are settled.

If you test button colours while your offer is unclear, you will get a two percent improvement on a page that was never going to work.

Change one thing, or accept you will not know why

If version B has a new headline, a new image and a shorter form, and it wins, what did you learn? That some combination of three changes helped. You cannot carry any of it to the next page with confidence.

There is a legitimate exception. If the current page is performing badly and you need a step change, rebuild it entirely and treat it as a fresh start rather than a test. Just be honest about which one you are doing. A rebuild answers "is this better", a test answers "why".

Let it run longer than feels comfortable

The single most common mistake is calling a winner too early. Small numbers swing wildly. Twelve conversions against eight looks decisive and means almost nothing.

Two rules that will keep you out of trouble.

Run for whole weeks, never partial ones. Your Tuesday traffic and your Saturday traffic behave differently, and a test that ran Thursday to Sunday is measuring the weekend, not the page.

Wait for enough conversions, not enough visitors. As a working minimum, aim for somewhere around a hundred conversions per version before you take the result seriously. If that will take three months at your traffic level, split testing is not your tool right now. Fix the offer and the traffic first, and come back to testing when the volume can support it.

Measure the thing that pays you

A page that gets more people into the funnel but fewer people to buy is not a winner, and plenty of tests are declared winners on exactly that basis.

Pick one primary metric and make it the closest thing to money you can measure. Booked calls, not clicks. Qualified enquiries, not form fills. If the test wins on the shallow metric and loses on the deep one, the losing version was actually right.

Watch the downstream numbers for a couple of weeks after you ship the winner. Occasionally a page wins by attracting a looser audience, and you only see the cost later in your close rate.

Keep a record, or repeat yourself forever

Write down every test in one place. What you changed, what you expected, what happened, and what you concluded.

Two lines per test is enough. Within a year you will have something more valuable than any of the individual results, which is a picture of what your particular audience responds to. That knowledge makes your next page better before you test anything at all.

Include the losers. A test that failed to move anything tells you that factor does not matter much to your buyers, and that is worth knowing.

When not to test

Testing is the wrong tool in three situations, and using it anyway wastes months.

When traffic is too low to reach a result before the market changes. Interview five customers instead. You will learn more.

When the page is fundamentally broken. Fix the obvious problems first, then test refinements. You do not need an experiment to tell you the mobile layout is unusable.

When you are not willing to act on the answer. If the winning version is one you would refuse to ship for brand reasons, you have not designed a test, you have designed a debate.

What good looks like

You make fewer changes and each one is more considered. You can point at your conversion rate over the last year and explain the jumps. You stop arguing about opinions in meetings, because the page settles it.

And you stop paying for traffic that lands on a page you have never seriously examined, which is the most expensive habit in this whole business.

What to do next

Look at the page that receives your most expensive traffic and write down the single biggest assumption it makes about your buyer. That assumption is your first real test, and it is almost never the button colour.

If you would rather have someone design the test properly, run it long enough, and read the result honestly, that is a good use of half an hour together.