Is my A/B test result significant? A plain-English guide with a worked example

October 10, 2026 · 5 min read

Version B of your signup page converted at 7% and version A at 5%. B looks like the winner. The question an A/B test has to answer is whether that gap is real, or whether you'd see one like it by chance even if both pages were identical. That's what statistical significance measures.

What "significant" means

Imagine A and B truly convert at the same rate. Visitors still arrive at random, so the two observed rates will rarely match exactly. The p-value is the chance of seeing a gap at least as big as yours under that assumption. A small p-value means "a gap this large would be unusual if nothing changed". The common threshold is 0.05, often described as 95% confidence.

It doesn't mean there's a 95% chance B is better, and it says nothing about how much better. It's a guard against fooling yourself with noise.

A worked example

Each version gets 1,000 visitors. A has 50 conversions (5%), B has 70 (7%).

Pooled rate: 120 conversions ÷ 2,000 visitors = 6%.
Standard error of the difference: √(0.06 × 0.94 × (1/1,000 + 1/1,000)) ≈ 0.0106.
z = (0.07 − 0.05) ÷ 0.0106 ≈ 1.88.
Two-sided p-value ≈ 0.06.

So a two-point lift, 40% in relative terms, is not significant at 95% with this much traffic. It's promising, and it might well be real, but a gap like this turns up by chance about 6 times in 100 when there's no difference. The A/B test significance calculator runs this same two-proportion z-test for your numbers.

Mistakes that create false winners

Stopping as soon as it looks significant. If you check every day and stop the first time p dips below 0.05, you'll declare far more false winners than 5%. Decide the sample size or duration in advance and check once.
Tiny samples. With a handful of conversions per version, the result swings wildly. The z-test's approximation is also unreliable below roughly 10 conversions per version; treat those results as a hint.
Changing several things at once. A new headline, button and price together can win, but you won't know which change did it.
Ignoring the calendar. Weekday and weekend visitors behave differently. Run tests for whole weeks so each version sees the same mix.
Unequal traffic. If one version got the newsletter blast and the other didn't, you're comparing audiences, not pages.

When you don't have much traffic

Small sites often can't reach significance on small changes in any reasonable time. That's useful to know: test bolder changes (a different offer, a much shorter form), where the effect is big enough to detect, rather than button colours. And measure the steps before the conversion, which have more traffic. A funnel shows which step loses the most people, and that's usually the best place for the next test.

To compare versions in measuremy.site, send each variant's conversion as an event with the variant as a property, and ask your coding agent for conversions by variant. Then put the numbers into the calculator before you call it.

See it on your own site

measuremy.site is cookieless analytics your coding agent can read. 30-day free trial, no card. Plans from $9/month.

Start free trial See the live demo

Add measuremy.site analytics to this project: connect the measuremy.site MCP server at https://a.measuremy.site/mcp (it signs in with OAuth, no key), then tell me how to sign in. Details: measuremy.site/llms.txt

Free tools

Free conversion rate calculator: Conversion rate from visitors and signups, plus the traffic needed for a goal.
Free A/B test significance calculator: Is your A/B test result real or luck? Check significance.
UTM link checker: Find broken UTM tags and see which GA4 channel a link lands in.

All free tools