Tools › Statistics › Hypothesis Tests › A/B Test › Example

A/B test of 5.0 % against 5.6 % conversion in 10,000 visitors each: z test, Newcombe interval, posterior probability and sample size

ToolA/B Test
SourceIllustrative case built on Newcombe, R. G. (1998). "Interval Estimation for the Difference Between Independent Proportions: Comparison of Eleven Methods." Statistics in Medicine, Vol. 17, No. 8, pp. 873–890 (method 10); Cook, J. D. (2005). Exact Calculation of Beta Inequalities. UTMDABTR-005-05, M. D. Anderson Cancer Center, Houston; R Core Team, stats::power.prop.test and prop.test; Kohavi, R., Longbotham, R., Sommerfield, D., and Henne, R. M. (2009). "Controlled Experiments on the Web: Survey and Practical Guide." Data Mining and Knowledge Discovery, Vol. 18, No. 1, pp. 140–181.
LevelProfessional

Problem

A web page is shown to 10,000 visitors in a control arm A, of whom 500 convert, and to 10,000 in a treatment arm B, of whom 560 convert. Is the treatment better, how large is the lift and how sure is it, and how many visitors would a test need to detect a 10 % relative lift (5.0 % to 5.5 %) with 80 % power at α = 0.05?

Inputs to Enter

  • Variants: A 10000 500 / B 10000 560 (conversion metric); confidence 0.95, two-sided, no multiple-comparison adjustment
  • Planning: MDE 10 % relative, power 0.80; prior Beta(1, 1); Monte Carlo seed 2026; 5 looks for the sequential caution
Open the A/B Test tool (loaded with this example)

Answers

  • Rates 5.00 % (Wilson 4.59 % to 5.44 %) and 5.60 % (5.17 % to 6.07 %); pooled z = 1.894, two-sided p = 0.0583; Fisher's exact p = 0.0625; R prop.test with Yates' correction X² = 3.468, p = 0.0626
  • Difference 0.60 points, Newcombe method 10 interval −0.021 to 1.222 points; relative uplift 12.0 %, log-ratio interval −0.40 % to 25.9 %
  • Posteriors Beta(501, 9501) and Beta(561, 9441): P(p_B > p_A) = 0.97084 by Cook's closed sum (scipy quadrature 0.9708377; 200,000 Monte Carlo draws with seed 2026: 0.9702); expected loss of choosing B 0.0036 % of conversion rate, of keeping A 0.603 %
  • Sample size per variant for 5.0 % → 5.5 % at 80 % power: 31,234 (R power.prop.test formula; Kohavi's 16 σ²/δ² rule 30,400); the 10,000 collected detect only an 18.0 % relative lift and had power 0.47 for the observed 12 % lift
  • Five unadjusted looks at α = 0.05 carry a 0.142 false-positive rate (Armitage, McPherson and Rowe 1969); Pocock constant 2.413, O'Brien–Fleming 2.040 (Jennison and Turnbull 2000)

Notes

The tool's default inputs are this case; press Submit. The counts are illustrative, not from a published experiment; every method's number is checked in test_ab_test.php against the sources named (Newcombe's Table II, Cook's closed forms, R's formulas, Kohavi's examples, the Jennison–Turnbull constants) and against scipy / numpy (References/AB-Test/check_ab.py).