Change one thing and know whether it worked

After launch, everyone has ideas for the site. Make five of them in one week and the numbers will move, and you'll never know which change did it, or whether any of them did.

So change one thing at a time, set up so you can tell what happened. Start from evidence, not a hunch: a question the numbers answer, the monthly search read, something a customer said. Write the hypothesis down. Make the one change. Compare a window before with a window after. Then record what you decided, including when the answer is 'we don't know'.

GOV.UK's data community teaches a format worth borrowing: if, then, because. Here it is for the bakery. On a phone, the big Order ahead button sits below the queue photo. So: if the button moves above the photo on phones, then more regulars will order ahead, because they order on their phones in the morning and may not scroll that far. Then one primary metric, orders ahead per week, chosen before the change goes live. Choose the metric afterwards and you'll always find one that went up. The change reverses Design's photo-first rule on phones, so it goes in the decision notes with its date.

An A/B test splits visitors at random between two versions, and it's the strongest way to know. It also needs a lot of visitors. A sample-size calculator tells you how many. You enter how often visitors do the thing now and the smallest change you'd care about, and it gives the number each version needs. If a site really has that traffic, run the test in a proper testing tool and use its statistics.

Here's the arithmetic, with invented numbers. Say two thousand visits a month reach a page and two percent do the thing you want. Even to detect a jump from two to three percent, which is a big one, the standard sample-size formula at the usual settings asks for more than three thousand visitors on each version. Split two thousand a month in half and one test takes over three months, with the season changing underneath it. Put your own numbers into a calculator. For most small sites the honest answer is no test, and the bakery's plan is a before and after.

A before-and-after window is weaker evidence, so say so, because other things change at the same time: the season, a newsletter, a competitor. Keep the two windows the same length and made of full weeks. GOV.UK suggests running even a proper test for at least a full week, to cover differences between days. Never use launch week as your before, because it's never a normal week. The bakery went live on 21 October, so its before window is the four full weeks after launch week. The change goes live on 23 November, dated in the log, and the after window is the four full weeks from then.

Sometimes the after window looks like the before, or the difference is tiny next to the normal wobble from week to week. That's a result too. GOV.UK makes the point that a properly designed test teaches you something whether or not there's a clear winner. Write 'no clear effect'. Decide whether the change stays for some other reason, because it reads more clearly, say. Then stop. Don't go through ten other metrics looking for one that moved.

The bakery's result isn't in yet, and nobody should guess it. So here's a finished entry from a fictional pottery studio, with made-up figures. Customers kept emailing to ask when the next class ran, so the studio moved the next date and price near the top of its class page. The entry gives the evidence, the metric, the two windows, and then a reading in plain words that admits what else happened. A newsletter went out during the after window, so the rise is encouraging and unproven. The decision is to keep the change and look again later. Someone reading this next year can see what you changed and how sure you were.

One change, a hypothesis first, a fair before and after, and a log of how sure you are.