Source: https://measuremy.site/blog/is-my-ab-test-significant [measuremy.site](https://measuremy.site/) / [Blog](https://measuremy.site/blog) # Is my A/B test result significant? A plain-English guide with a worked example October 10, 2026 · 5 min read Version B of your signup page converted at 7% and version A at 5%. B looks like the winner. The question an A/B test has to answer is whether that gap is real, or whether you'd see one like it by chance even if both pages were identical. That's what statistical significance measures. ## What "significant" means Imagine A and B truly convert at the same rate. Visitors still arrive at random, so the two observed rates will rarely match exactly. The p-value is the chance of seeing a gap at least as big as yours under that assumption. A small p-value means "a gap this large would be unusual if nothing changed". The common threshold is 0.05, often described as 95% confidence. It doesn't mean there's a 95% chance B is better, and it says nothing about how much better. It's a guard against fooling yourself with noise. ## A worked example Each version gets 1,000 visitors. A has 50 conversions (5%), B has 70 (7%). - Pooled rate: 120 conversions ÷ 2,000 visitors = 6%. - Standard error of the difference: √(0.06 × 0.94 × (1/1,000 + 1/1,000)) ≈ 0.0106. - z = (0.07 − 0.05) ÷ 0.0106 ≈ 1.88. - Two-sided p-value ≈ 0.06. So a two-point lift, 40% in relative terms, is **not** significant at 95% with this much traffic. It's promising, and it might well be real, but a gap like this turns up by chance about 6 times in 100 when there's no difference. The [A/B test significance calculator](https://measuremy.site/tools/ab-test-significance-calculator) runs this same two-proportion z-test for your numbers. ## Mistakes that create false winners - **Stopping as soon as it looks significant.** If you check every day and stop the first time p dips below 0.05, you'll declare far more false winners than 5%. Decide the sample size or duration in advance and check once. - **Tiny samples.** With a handful of conversions per version, the result swings wildly. The z-test's approximation is also unreliable below roughly 10 conversions per version; treat those results as a hint. - **Changing several things at once.** A new headline, button and price together can win, but you won't know which change did it. - **Ignoring the calendar.** Weekday and weekend visitors behave differently. Run tests for whole weeks so each version sees the same mix. - **Unequal traffic.** If one version got the newsletter blast and the other didn't, you're comparing audiences, not pages. ## When you don't have much traffic Small sites often can't reach significance on small changes in any reasonable time. That's useful to know: test bolder changes (a different offer, a much shorter form), where the effect is big enough to detect, rather than button colours. And measure the steps before the conversion, which have more traffic. A [funnel](https://measuremy.site/blog/conversion-funnel-analysis) shows which step loses the most people, and that's usually the best place for the next test. To compare versions in measuremy.site, send each variant's conversion as an event with the variant as a property, and ask your coding agent for conversions by variant. Then put the numbers into the calculator before you call it. ### See it on your own site measuremy.site is cookieless analytics your coding agent can read. 30-day free trial, no card. Plans from $9/month. [Start free trial](https://measuremy.site/login) [See the live demo](https://measuremy.site/share/measuremy.site) Add measuremy.site analytics to this project: connect the measuremy.site MCP server at https://a.measuremy.site/mcp (it signs in with OAuth, no key), then tell me how to sign in. Details: measuremy.site/llms.txt ## Free tools - [Free conversion rate calculator](https://measuremy.site/tools/conversion-rate-calculator): Conversion rate from visitors and signups, plus the traffic needed for a goal. - [Free A/B test significance calculator](https://measuremy.site/tools/ab-test-significance-calculator): Is your A/B test result real or luck? Check significance. - [UTM link checker](https://measuremy.site/tools/utm-link-checker): Find broken UTM tags and see which GA4 channel a link lands in. [All free tools](https://measuremy.site/tools) ## More from the blog - [How to calculate your website conversion rate, and what to compare it to](https://measuremy.site/blog/calculate-website-conversion-rate) - [What is bounce rate, how is it calculated, and is yours a problem?](https://measuremy.site/blog/what-is-bounce-rate) - [How to add a visitor counter to your website or GitHub README](https://measuremy.site/blog/add-visitor-counter-to-website) - [How much traffic does a website get? How to estimate any site's visits](https://measuremy.site/blog/how-much-traffic-does-a-website-get) - [How to track QR code scans: count them, and see which poster worked](https://measuremy.site/blog/track-qr-code-scans) - [How to see how many people clicked your link (without a paid shortener)](https://measuremy.site/blog/how-many-people-clicked-my-link) - [How to check if your Google Analytics 4 tag is installed (and only counting once)](https://measuremy.site/blog/check-if-ga4-is-installed) - [Why GA4 shows your campaign traffic as Unassigned (and how to fix the links)](https://measuremy.site/blog/ga4-unassigned-traffic) - [Does your redirect keep your UTM tags? How to check before a campaign goes out](https://measuremy.site/blog/redirect-drops-utm-tags) - [How I built a self-hosted analytics MCP server on Cloudflare Workers and D1](https://measuremy.site/blog/analytics-mcp-server-cloudflare-workers) - [UTM tracking explained: what the parameters mean and how to use them](https://measuremy.site/blog/utm-tracking-explained) - [Conversion funnel analysis for small sites: a practical guide](https://measuremy.site/blog/conversion-funnel-analysis) - [Sudden drop in website traffic? A checklist for finding the cause](https://measuremy.site/blog/sudden-drop-in-website-traffic) - [What is an MCP server? A plain-English guide for people who run websites](https://measuremy.site/blog/what-is-an-mcp-server) - [What Is the Urchin Tracking Module?](https://measuremy.site/blog/what-is-the-urchin-tracking-module-and-why-utm-stuck-around) - [What Microsoft MCP servers actually do](https://measuremy.site/blog/microsoft-mcp-servers-explained) - [UTM parameters for email newsletters](https://measuremy.site/blog/utm-parameters-for-email-newsletters) - [Track Button Clicks Without Google Tag Manager](https://measuremy.site/blog/how-to-track-clicks-on-a-button-without-google-tag-manager) - [How to track the results of a social media post without cookies](https://measuremy.site/blog/how-to-measure-the-results-of-a-social-media-post) - [How to track traffic from a GitHub README](https://measuremy.site/blog/how-to-track-traffic-from-a-github-readme) - [How to measure a product launch without guessing](https://measuremy.site/blog/how-to-measure-a-product-launch)