A/B Test

A/B Test

A/B testing: frequentist, Bayesian, planning and peeking

What an A/B test is

An online controlled experiment assigns each visitor at random to a control (A, the current page) or to one or more treatments (B, C, …, the candidate changes) and compares a metric between the arms: a conversion rate (the share of visitors who click, sign up or buy) or a continuous quantity (revenue per visitor, time on page). Randomization makes the arms exchangeable, so a difference in the metric is attributed to the change and not to the kind of visitor who happened to see it. The analysis has to answer three questions: is the difference more than chance would produce, how large is it likely to be, and was the test big enough to see the effect that matters. This page gives the methods behind each section of the tool. Kohavi, Tang and Xu (2020) is the reference text for the practice; the formulas below are standard and the sources of each are named.

Rates and their intervals

A conversion rate is a binomial proportion p̂ = x/n. Its confidence interval is the Wilson (1927) score interval, ((p̂ + z²/2n) ± z √(p̂(1 − p̂)/n + z²/4n²)) / (1 + z²/n) with z the (1 − α/2) normal quantile. Unlike the textbook Wald interval p̂ ± z √(p̂(1 − p̂)/n) it never leaves [0, 1], has a sensible width when x is 0 or n, and keeps its coverage near the nominal level for the small rates of online conversion (Newcombe 1998, single-proportion paper). For a continuous metric the tool reports the mean, the sample standard deviation s and the t interval m ± t_{1 − α/2, n − 1} s/√n.

Frequentist comparison

Two-proportion z test. Under the null hypothesis p_B = p_A both arms estimate the same rate, so the standard error uses the pooled proportion p̄ = (x_A + x_B)/(n_A + n_B): z = (p̂_B − p̂_A) / √(p̄(1 − p̄)(1/n_A + 1/n_B)). Its square is Pearson's χ² on the 2 × 2 table, and the two-sided p-value is P(|Z| ≥ |z|). R's prop.test applies Yates' continuity correction by default (subtracting ½ from each |O − E|), which this tool does not; with online sample sizes the two agree to the third decimal, and for small counts Fisher's exact test is the one to quote. The one-sided alternative (treatment better) halves the p-value when the treatment is ahead; the two-sided test is the usual choice because a treatment that is worse must be found too.

Difference of rates. The interval for D = p̂_B − p̂_A is Newcombe's (1998) method 10, the hybrid score interval built from the two Wilson intervals (l_A, u_A) and (l_B, u_B): lower bound D − √((p̂_B − l_B)² + (u_A − p̂_A)²), upper bound D + √((u_B − p̂_B)² + (p̂_A − l_A)²). In Newcombe's comparison of eleven methods it was among the best for coverage and width at every sample size, and it is defined when a count is 0 or equals n, where the Wald interval collapses to a point.

Relative uplift. Product teams read the result as a relative change, p̂_B/p̂_A − 1. Its interval comes from the large-sample variance of the log ratio, Var[ln(p̂_B/p̂_A)] = (1 − p̂_A)/(n_A p̂_A) + (1 − p̂_B)/(n_B p̂_B) (the delta method applied to the ratio of two independent binomial proportions); the bounds are exp(ln(p̂_B/p̂_A) ± z SE) − 1. The interval is asymmetric about the point estimate, as a ratio should be, and requires both rates to be positive.

Fisher's exact test conditions on the margins of the 2 × 2 table; the two-sided p-value sums the hypergeometric probabilities of every table no more probable than the observed one (R's fisher.test convention). It is exact for any counts and the one to report when a cell is below about 5.

Several variants. Pearson's χ² test of homogeneity on the k × 2 table, with expected counts n_i p̄ and n_i(1 − p̄) and k − 1 degrees of freedom, asks whether the rates differ at all. Each treatment is also compared with the control, and because several comparisons share one control the family-wise false-positive rate grows with their number. Bonferroni multiplies every p-value by the number of comparisons m; Holm's (1979) step-down procedure orders the p-values, multiplies the smallest by m, the next by m − 1 and so on, enforces monotonicity, and controls the same family-wise error with more power. Treatments compared with one another, rather than with the control, are not tested.

Continuous metrics. Welch's (1947) t test compares two means without assuming equal variances: t = (m_B − m_A) / √(s_A²/n_A + s_B²/n_B) on the Welch–Satterthwaite degrees of freedom. Cohen's d = (m_B − m_A)/s_p with the pooled standard deviation puts the difference in standardized units (0.2 small, 0.5 medium, 0.8 large; Cohen 1988). Revenue-type metrics are heavily skewed; Kohavi, Deng, Longbotham and Xu (2014) recommend at least 355 s² observations per arm, s being the skewness, before the mean is treated as normal, and capping the metric to reduce the skewness.

Bayesian analysis

With a Be⁠ta(a, b) prior on a conversion rate, the posterior after x conversions in n visitors is Be⁠ta(a + x, b + n − x): the prior counts as a conversions in a + b visitors, and Be⁠ta(1, 1) is the flat prior. The quantity a product team wants is the probability that the treatment's true rate exceeds the control's, P(p_B > p_A) = ∫₀¹ f_A(x) [1 − F_B(x)] dx with f_A the posterior density of A and F_B the posterior distribution function of B. Cook (2005) showed that this Be⁠ta inequality has a closed form whenever one of the four parameters is a positive integer, which is always the case with an integer prior: in the form Evan Miller (2015) gives it, P(p_B > p_A) = Σ_i=0^{a_B − 1} B(a_A + i, b_A + b_B) / [(b_B + i) B(1 + i, b_B) B(a_A, b_A)], evaluated in logarithms of the Be⁠ta function. The tool computes it exactly this way (choosing the smallest integer parameter among the four equivalent sums), by Gauss–Legendre quadrature when no parameter is an integer, and again by 200,000 Monte Carlo draws from the posteriors with a fixed seed, so the two agree to about three decimals and the simulation also gives P(best) when there are several treatments.

The expected loss of choosing a variant is E[max(p_other − p_chosen, 0)], the average conversion rate given up if the other variant is in fact better; it is small when the chosen variant is very likely better or when the two are nearly equal, which makes it a decision quantity rather than a significance one. Stucchio (2015) proposed stopping when the expected loss of the leading variant falls below a threshold of caring, such as 0.1 % of conversion rate. The tool evaluates it exactly for each treatment against the control by quadrature and for every variant against the best by Monte Carlo. The highest-density interval of the uplift is the shortest interval holding the chosen share of the posterior draws of p_B/p_A − 1 (and of the difference); unlike the equal-tailed interval it is centred on the most probable values.

The Bayesian numbers do not suffer the peeking problem in the sense that P(p_B > p_A) is a valid posterior statement at any time, but a rule such as "ship when P > 0.95" examined repeatedly still ships flat changes more often than 5 % of the time; the expected-loss rule degrades more gracefully. The Bayesian section applies to conversion metrics; a continuous metric's frequentist interval is its report (a normal posterior with a flat prior would reproduce it).

Planning: sample size, duration, detectable effect

Before the test, fix the smallest effect worth detecting (the minimum detectable effect, MDE, relative to the control or in absolute percentage points), the power 1 − β (usually 0.8) and α. The sample size per variant for two proportions is the formula of R's power.prop.test (strict = FALSE), n = (z_{1 − α/2} √(2 p̄ q̄) + z_{1 − β} √(p₁q₁ + p₂q₂))² / (p₂ − p₁)² with p̄ = (p₁ + p₂)/2: the pooled variance under the null in the critical term and the two separate variances under the alternative (Fleiss, Levin and Paik 2003 give the same form, with an optional continuity correction the tool does not apply). Kohavi, Longbotham, Sommerfield and Henne (2009) reduce it to the rule of thumb n = 16 σ²/δ² per variant for 80 % power at 95 % confidence (21 σ²/δ² for 90 %), with σ² = p(1 − p) for a rate and δ the absolute change; the 16 is (z_0.975 + z_0.8)² × 2 ≈ 15.7 rounded up. Their example, a 5 % conversion rate and a 5 % relative change, needs 16 × 0.0475 / 0.0025² = 121,600 visitors per variant. For a continuous metric n = 2 (z_{1 − α/2} + z_{1 − β})² σ²/δ². Evan Miller's sample-size calculator uses the same two-term form but with the baseline variance 2p₁(1 − p₁) in the null term instead of 2 p̄ q̄, and gives 1,030 per variation for a 20 % baseline and a 5-point absolute effect at 80 % power where R's formula gives 1,094; the tool reports R's.

The duration is the total sample over all variants divided by the daily traffic entering the test; run whole weeks so weekday and weekend visitors are both represented. The tool also solves the same power equation the other way: the effect the current sample detects at the target power, and the power the test had for the effect it observed. The latter, the "observed power", is a function of the p-value and adds no information; it is shown because it is asked for.

The peeking problem and sequential boundaries

A p-value is valid for one analysis at a predetermined sample size. If the dashboard is watched and the test stopped the first time p drops below 0.05, the chance of a false positive is no longer 5 %: Armitage, McPherson and Rowe (1969) computed it as 0.083 for two looks, 0.142 for five and 0.193 for ten equally spaced looks, and it tends to 1 with unlimited looking. The remedy is either to fix the sample size in advance or to use a group-sequential design in which the critical value at each look is raised so the total α is held. Pocock (1977) uses one constant C_P at every look (|Z_k| ≥ C_P); O'Brien and Fleming (1979) use C_B √(K/k) at look k of K, very strict early and close to the fixed-sample 1.96 at the end, which is why it is the common choice. The constants for K = 1 to 10 equally spaced looks at two-sided α = 0.05 and 0.01 are tabulated by Jennison and Turnbull (2000, Tables 2.1 and 2.3): for five looks C_P = 2.413 and C_B = 2.040, so the first O'Brien–Fleming look needs |z| ≥ 2.040 √5 = 4.56. The tool lists the boundary of every look with its nominal p-value as a caution; a real group-sequential analysis fixes K and the schedule beforehand and derives its final interval and p-value from the stopping rule.

Reading the results

Report the rates with their intervals, the difference and the relative uplift with theirs, the p-value with the alternative stated, and the posterior probability with the expected loss: a treatment with p = 0.06, P(better) = 0.97 and an expected loss of 0.004 % of conversion rate is not "a failure to reach significance" but a change that is very probably positive and very cheap to ship, whereas the same probability with a wide uplift interval says the size of the gain is still unknown. Check the planning section against the sample actually collected: a test that could only detect an 18 % relative lift was never going to establish a 5 % one, and a non-significant result from it is uninformative rather than negative. Finally, the analysis assumes the randomization worked (sample-ratio mismatch between the arms is the first thing to check), that visitors are independent (one user counted once), and that the metric was fixed before the data were seen.

Worked example: 500 against 560 conversions in 10,000 visitors each

The default case is illustrative. The control converts 5.00 % (Wilson 4.59 % to 5.45 %), the treatment 5.60 % (5.17 % to 6.07 %). Pooled z = 0.006/0.003168 = 1.894, two-sided p = 0.0583 (Fisher 0.0625; R's prop.test with Yates' correction gives X² = 3.468, p = 0.0626): not significant at 0.05. The difference is 0.60 points with Newcombe interval −0.02 to 1.22 points, the relative uplift 12.0 % with interval −0.4 % to 25.9 %. With a flat prior the posteriors are Be⁠ta(501, 9501) and Be⁠ta(561, 9441); P(p_B > p_A) = 0.9708 by the closed sum, the expected loss of shipping B is 0.0036 % of conversion rate against 0.60 % for keeping A. To detect a 10 % relative lift (5.0 % to 5.5 %) at 80 % power the test needs 31,234 visitors per variant (Kohavi's rule: 30,400); the 10,000 collected detect only an 18 % lift, and the power at the observed 12 % lift was 0.47. Five unadjusted looks would carry a 14.3 % false-positive rate; the O'Brien–Fleming boundary at the final look is 2.040 and the observed |z| of 1.894 does not reach it.