Sample Size and Power

Sample Size and Power

Sample Size, Power and the Detectable Effect

What the tool does

Every test has two error rates: α, the probability of rejecting a true null hypothesis, and β, the probability of missing a real effect; 1 − β is the power. Fix three of {sample size, effect size, α, power} and the fourth follows. The tool solves whichever of the three is asked for (the sample size for a given effect and target power, the power reached by a given sample, or the smallest effect a given sample detects at the target power) for the designs listed on the page, prints the formula with the substituted numbers and the distribution behind it, and draws the power against the sample size and against the effect size so the trade-off is visible. A table gives the sample size at 70, 80, 90 and 95 % power for three significance levels. The sample sizes are integers rounded up; where a well-known program prints the continuous root of the power equation (R's power.t.test, power.prop.test, power.anova.test), that root is printed as well so the two can be compared digit for digit.

Conventions

The significance level α is two-sided when the alternative is two-sided and one-sided otherwise, except for non-inferiority and equivalence, which are one-sided tests at α each (0.025 is the conventional value). The power of a two-sided test counts rejections in either direction; R's power functions count only the direction of the true effect (their strict = FALSE default), which differs by the probability of the opposite tail, below 10⁻⁶ at usual powers, and both numbers are printed for the means and proportions. The allocation ratio κ = n₂/n₁ sets the second group at κ times the first, rounded up. The direction of a one-sided alternative refers to group 1 minus group 2 (or the sample against the null value). Effect sizes are entered in the units of the measurement (a difference δ with its standard deviation σ) or as the dimensionless measures of Cohen (1988): d = δ/σ, h for proportions, r, f for ANOVA and w for χ²; the page names Cohen's small, medium and large conventions for the measure in use as a last resort when no pilot estimate exists.

Means: one sample, paired, two samples

With σ known the statistic is normal, the sample size is N = (z_1−α/2 + z_1−β)² (σ/δ)² for one sample (NIST e-Handbook 7.2.2.2) and n₁ = (z_1−α/2 + z_1−β)² (σ₁² + σ₂²/κ)/δ² for two, and the power is exact. With σ estimated the t statistic has, under the alternative, the noncentral t distribution with noncentrality λ = δ/SE and ν degrees of freedom (n − 1; n₁ + n₂ − 2 pooled; the Welch–Satterthwaite value with the population variances for unequal variances), and the power is 1 − F_ν,λ(t_c) + F_ν,λ(−t_c) (Owen 1965). The tool evaluates the noncentral t by Lenth's AS 243 algorithm, the one R uses, and finds the smallest integer n at which the power reaches the target; the normal formula is printed beside it and is always a little low for a t test because the t critical value exceeds z. The e-Handbook's iteration with t critical values ((1.860 + 1.397)² = 10.6 for its example) is the hand version of the same calculation. A paired design is a one-sample design on the differences, with σ the standard deviation of a difference.

Proportions

One proportion: the z test of p̂ against p₀ with the variance under the alternative gives N ≥ [(z_1−α √(p₀q₀) + z_1−β √(p₁q₁)) / |p₁ − p₀|]² (NIST e-Handbook 7.2.4.2, after Fleiss, Levin and Paik 2003), and the continuity correction adds 1/|p₁ − p₀|. The exact binomial test rejects when the count falls in the tail whose probability under p₀ does not exceed α (α/2 each side when two-sided); its power is the binomial probability of that region under p₁ and is a sawtooth in n, so the page lists the sizes just above the first crossing at which the power dips below the target again. Two proportions: the pooled form, the one R's power.prop.test uses and Fleiss's eq. 4.14 inverts, has the null variance pooled over the two groups and the alternative variance unpooled; the continuity-corrected size is Fleiss's eq. 4.15 generalized to unequal allocation, n₁′ = n₁/4 [1 + √(1 + 2(κ + 1)/(κ n₁ |p₁ − p₂|))]²; the arcsine form uses Cohen's h = 2 asin √p₁ − 2 asin √p₂, whose variance 1/n₁ + 1/n₂ does not depend on the proportions, so n₁ = ((z_1−α/2 + z_1−β)/h)² (1 + 1/κ). Fisher's exact test conditions on the total number of successes: for every total the tool finds the hypergeometric rejection region at level α (two-sided: the tables as or less probable than the observed one, the rule of R's fisher.test) and sums the binomial probabilities of all rejecting tables; the exact test is conservative, so it needs more subjects than the z test (64 against 58 per group for 0.50 against 0.75 at 80 % power).

Correlation, ANOVA, χ², regression slope, variance

A correlation is tested through Fisher's z = atanh r, normal with variance 1/(n − 3): n = [(z_1−α/2 + z_1−β)/(atanh r − atanh r₀)]² + 3. A balanced one-way ANOVA with k groups of n has a noncentral F distribution with λ = k n f², f = σ_means/σ; R's power.anova.test takes the between-group variance var(μ_i) instead, so f² = (k − 1)/k · between.var/within.var, and the tool reproduces its printed values (0.3535594 for four groups of five with variances 1 and 3). Pearson's χ² on N observations has a noncentral χ² distribution with λ = N w². The slope of a simple regression is tested by t with n − 2 degrees of freedom and noncentrality β₁ σ_x √n/σ_ε (Dupont and Plummer 1998), the same as the noncentral F(1, n − 2) with λ = n f². The one-sample variance test follows the e-Handbook's 7.2.3.2 procedure (Diamond 1989): if the true variance is R σ₀², the statistic (n − 1)s²/σ₀² is R times a χ²_n−1 variable and the power is P(χ²_n−1 > χ²_1−α,n−1/R); the handbook's example (R = 1.55, α = 0.05, β = 0.01) has its root at 169.3 degrees of freedom. The noncentral χ² and F are evaluated as Poisson mixtures of the central distributions (Johnson, Kotz and Balakrishnan 1995) and agree with scipy to 10⁻¹¹.

Survival

The log-rank test's power depends on the number of events d, not the number of patients: Schoenfeld (1983) gives d = (z_1−α/2 + z_1−β)² / (q₁ q₂ ln²HR), q₁, q₂ the allocation fractions; Freedman (1982) gives d = (z_1−α/2 + z_1−β)² (q₁ + q₂ HR)² / (q₁ q₂ (1 − HR)²), which reduces to (1 + HR)²/(1 − HR)² times the squared z sum for equal groups and is a little larger than Schoenfeld's for HR below 1. Both are printed; the Schoenfeld count is carried forward. The number of patients follows from the probability that a patient has an event before the analysis: with exponential survival (hazard λ = ln 2 / median), accrual uniform over t_a and follow-up t_f after the last entry, P = 1 − [e^−λt_f − e^{−λ(t_a+t_f)}]/(λ t_a) in each arm, and N = d/(q₁P₁ + q₂P₂). Loss to follow-up, non-proportional hazards and interim analyses are not modelled.

Non-inferiority and equivalence

Non-inferiority tests H₀: μ_T − μ_R ≤ −m against the one-sided alternative; with true difference ε the noncentrality is (ε + m)/SE and n_T = (z_1−α + z_1−β)² σ² (1 + 1/κ)/(ε + m)² (Chow, Shao and Wang 2008 §3.2.2; the proportion form §4.2.2 replaces σ²(1 + 1/κ) by p_Tq_T + p_Rq_R/κ). The tool reports the exact noncentral-t power for means. Equivalence uses Schuirmann's two one-sided tests: both H₀₁: difference ≤ −m and H₀₂: difference ≥ m must be rejected at level α, equivalent to the (1 − 2α) confidence interval lying inside ± m. The usual approximation n_T = (z_1−α + z_1−β/2)² σ² (1 + 1/κ)/(m − |ε|)² is printed, and the power reported for means is exact: conditional on the sample standard deviation s, both tests reject together when −m + t_1−α,ν s c < D̄ < m − t_1−α,ν s c (c = √(1/n_T + 1/n_R)), an interval that is empty when s is large, and the normal probability of that interval is integrated over the distribution of s (Owen's Q function, evaluated by Gauss–Legendre quadrature and checked against scipy and a Monte Carlo). The exact value is below the sum-of-two-normal-probabilities approximation at small n (0.0007 against a meaningful figure at 20 per group for the page's example) and the two agree within a few thousandths above about 100 per group.

Cluster designs and surveys

When individuals are randomized in clusters, observations within a cluster are correlated and the variance of a cluster mean of m individuals is inflated by the design effect DE = 1 + (m − 1)ρ, ρ the intraclass correlation (Kish 1965; Donner and Klar 2000). The individual-level sample size per arm (from the two-sample t or two-proportion test, or entered) is multiplied by DE and divided by m to give the clusters per arm; with few clusters one more per arm is prudent for the cluster-level degrees of freedom. The survey design gives the simple-random-sample size for a margin of error e at confidence 1 − α: n₀ = z² p(1 − p)/e² for a proportion (Cochran 1977 §4.4; 384 for ± 5 points at 95 % with p = 0.5) or (zσ/e)² for a mean, reduced for a finite population to n₀/(1 + (n₀ − 1)/N); the margin of a planned n is reported with the finite-population factor √((N − n)/(N − 1)). With strata (N_h, S_h) the page allocates the sample proportionally, n_h = n N_h/N, and by Neyman's rule, n_h ∝ N_h S_h, which minimizes V(ȳ_st) = Σ W_h² S_h² (1/n_h − 1/N_h); for a survey mean it also gives the total n for the target margin under each allocation (Cochran eqs. 5.25 and 5.26). Unequal weighting adds its own design effect 1 + CV²(w), handled in the Survey Analysis tool.

Worked example on the page

The default case is the two-sample t test for a standardized difference of 0.5 at α = 0.05 (two-sided) and 80 % power. The normal formula gives 62.8 per group; the exact noncentral t needs 64 per group (power 0.8015; 63 gives 0.7952), and R's power.t.test(delta = 0.5, power = 0.8) prints the continuous root 63.77. The test script reproduces every printed example of R's documentation pages (power 0.740 and n = 76.7 for 0.50 against 0.75, p₂ = 0.8026 at n = 50, n = 73.37 for 0.90 against 1.0, the ANOVA values 0.3535594, 11.92613 and 15.18834), the NIST e-Handbook examples of 7.2.2.2, 7.2.3.2 and 7.2.4.2, and an independent numpy/scipy script (noncentral distributions on 527 grid points, Fisher's exact power by brute force, the exact TOST power by quadrature and Monte Carlo, Neyman allocation against random allocations, 20 seeded cases).

Limits of the tool

• Balanced ANOVA only (one factor, equal groups); no repeated measures, covariates or multiple comparisons.
• The correlation uses the Fisher-z approximation; the slope test assumes a fixed design with the entered spread of x.
• Survival assumes exponential survival, proportional hazards, uniform accrual, no dropout and a single analysis.
• Equivalence for proportions uses the normal approximation with unpooled variances; the exact TOST is for means.
• Fisher's exact power is enumerated up to 300 per group (400 when the power at a given n is asked for); beyond that the pooled z value stands in.
• The cluster design uses the design effect alone, not a cluster-level t test with its reduced degrees of freedom.
• No adjustment for multiple endpoints, interim looks, non-response or attrition: inflate the sizes for those as the protocol requires.