Hypothesis Tests: Method

Hypothesis tests, p-values, effect sizes and power

The logic of a test

A hypothesis test asks whether the data are consistent with a null hypothesis H₀ (no difference, no effect, independence). A test statistic is computed from the data whose sampling distribution under H₀ is known: t, z, χ² or F. The p-value is the probability, if H₀ were true, of a statistic at least as extreme as the one observed; the critical value is the point of that distribution that cuts off the tail area α. The two views agree: p < α exactly when the statistic lies beyond the critical value. The decision is to reject H₀ at level α or not; failing to reject is not evidence that H₀ is true, only that the data do not contradict it at the sample size in hand.

The alternative hypothesis sets which tail counts. A two-sided alternative (μ ≠ μ₀) rejects for large values of |t| and the p-value is the area in both tails; a one-sided alternative (μ < μ₀ or μ > μ₀) rejects in one tail only, with a smaller critical value, and must be chosen before the data are seen. The confidence interval at level 1 − α is the set of hypothesized values that the test would not reject: a two-sided 95 % interval that excludes μ₀ says the same thing as p < 0.05.

Tests of means

One-sample t (Student 1908): t = (m − μ₀) / (s/√n) on n − 1 degrees of freedom, where m and s are the sample mean and standard deviation. It assumes a random sample from a normal population; by the central limit theorem the mean is close to normal for samples of a few dozen from most distributions, so the test is robust except for small samples from skewed or heavy-tailed populations.

Paired t: when each observation in one condition is matched to one in the other (the same patient under two drugs, the same specimen before and after), the differences d_i are a single sample and the one-sample t test is applied to them. The pairing removes the variation between subjects, which is why Student's sleep data give p = 0.003 paired but p = 0.08 as two independent samples.

Two-sample t, pooled: with equal population variances the two sample variances are pooled, s_p² = ((n₁ − 1) s₁² + (n₂ − 1) s₂²) / (n₁ + n₂ − 2), and t = (m₁ − m₂ − D₀) / (s_p √(1/n₁ + 1/n₂)) on n₁ + n₂ − 2 degrees of freedom.

Two-sample t, Welch (Welch 1947): without the equal-variance assumption the standard error is √(s₁²/n₁ + s₂²/n₂) and the degrees of freedom are the Welch-Satterthwaite value df = (s₁²/n₁ + s₂²/n₂)² / ((s₁²/n₁)²/(n₁ − 1) + (s₂²/n₂)²/(n₂ − 1)), a fraction between the smaller n − 1 and n₁ + n₂ − 2. The two tests give the same t when n₁ = n₂ and nearly the same p when the variances are similar; when the variances and the sizes both differ the pooled test can be badly wrong in either direction, so Welch's test is the one to report unless equal variances are known from the measurement process. The tool reports both and the variance ratio.

Tests of proportions

One proportion: x successes in N trials give p̂ = x/N; under H₀: p = p₀ the statistic z = (p̂ − p₀) / √(p₀(1 − p₀)/N) is approximately standard normal when N p₀ and N (1 − p₀) both exceed about 5. The exact binomial p-value needs no approximation: for a one-sided alternative it is the binomial tail, for a two-sided one the sum of the probabilities of every outcome no more probable than the observed count (the convention of R's binom.test). The Wald interval p̂ ∓ z √(p̂(1 − p̂)/N) is the textbook one but misbehaves near 0 or 1; the Wilson (1927) score interval, which inverts the test itself, is reported as the one to quote.

Two proportions: under H₀: p₁ = p₂ both samples estimate the same proportion, so the test uses the pooled p̄ = (x₁ + x₂)/(N₁ + N₂) in the standard error, z = (p̂₁ − p̂₂) / √(p̄(1 − p̄)(1/N₁ + 1/N₂)); z² is Pearson's χ² for the 2 × 2 table. The confidence interval for the difference uses the unpooled standard error √(p̂₁(1 − p̂₁)/N₁ + p̂₂(1 − p̂₂)/N₂), since the two proportions need not be equal. No continuity correction is applied; with a cell below 5 use Fisher's exact test instead.

Chi-square tests

Goodness of fit: for k categories with observed counts O_i and expected counts E_i (from a hypothesized set of proportions, scaled to the total), χ² = Σ (O_i − E_i)²/E_i is approximately chi-square on k − 1 degrees of freedom. Independence: for an r × c table the expected count of a cell under independence is (row total)(column total)/N and χ² has (r − 1)(c − 1) degrees of freedom. Both tests are right-tailed: only a large χ² contradicts H₀. The approximation wants every expected count of 5 or more (the tool notes when one is smaller); the tool applies no Yates correction, so its 2 × 2 result equals the two-proportion z test squared.

One-way ANOVA

For k independent groups the total sum of squares splits into the part between the group means, SS_tr = Σ n_i(ȳ_i − ȳ)², and the part within the groups, SS_E; the mean squares are the sums divided by their degrees of freedom k − 1 and N − k, and F = MS_tr/MS_E is compared with the F distribution (Montgomery 2013, ch. 3). The assumptions are independent normal groups with a common variance; the tool flags a ratio of the largest to the smallest group standard deviation above 2. A significant F says only that the means are not all equal; the Tukey HSD procedure then compares every pair while keeping the family-wise error rate at α: a pair differs when |ȳ_i − ȳ_j| exceeds q_α(k, N − k) √(MS_E/2 (1/n_i + 1/n_j)), the Tukey-Kramer form, which reduces to q √(MS_E/n) for equal sizes (Tukey 1949). The tool computes the studentized range distribution of q by numerical integration rather than from a table; the Bonferroni t (critical t at α/(2m) for m comparisons) is shown beside it and is slightly more conservative.

Effect sizes

A p-value depends on the sample size as much as on the effect: a trivial difference is significant with enough data and a large one is not with too little. The effect size measures the difference itself in standardized units (Cohen 1988). Cohen's d = (m₁ − m₂)/s_p for two samples, (m − μ₀)/s for one, and the mean difference over its standard deviation for pairs (d_z); d is biased upward in small samples, and Hedges' g = J d with J = Γ(df/2) / (√(df/2) Γ((df − 1)/2)) ≈ 1 − 3/(4 df − 1) corrects it (Hedges 1981). Cohen's h = 2 asin √p₁ − 2 asin √p₂ puts proportions on the arcsine scale, where a given difference carries the same power whatever the baseline. Cramér's V = √(χ²/(N (k − 1))), k the smaller table dimension, runs from 0 to 1 and equals φ for a 2 × 2 table; Cohen's w = √(χ²/N) is the planning quantity. For ANOVA η² = SS_tr/SS_T is the share of the variation explained by the groups, ω² a less biased version, and Cohen's f = √(η²/(1 − η²)) the planning quantity. Cohen's conventions, small / medium / large: d and h 0.2 / 0.5 / 0.8, w 0.1 / 0.3 / 0.5, f 0.10 / 0.25 / 0.40. They are rules of thumb from the behavioral sciences; in a field with its own benchmarks use those.

Power and sample size

The power of a test is the probability of rejecting H₀ when a given alternative is true, 1 − β. For the t tests the statistic under the alternative follows the noncentral t distribution with noncentrality δ = d √n (one sample, pairs) or d √(n₁ n₂/(n₁ + n₂)) (two samples), and the power is the probability that it falls beyond the critical value; the tool evaluates it with Lenth's algorithm AS 243. The sample size for a target effect and power is the smallest n whose power reaches the target, found by iteration because the critical value moves with the degrees of freedom; the normal approximation n = (z_{1 − α/2} + z_{1 − β})²/d² (twice that per group for two samples) runs one or two short of the exact value. Cohen's reference values: d = 0.5 at power 0.80, α = 0.05 two-sided, needs 64 per group for two samples and 34 for one sample or pairs.

For proportions the normal approximation gives n = ((z_{1 − α/2} + z_{1 − β})/h)² per group from Cohen's h, independent of the baseline. Fleiss, Levin and Paik (2003) work on the proportion scale: n = (z_α √(2 p̄ q̄) + z_β √(p₁ q₁ + p₂ q₂))² / (p₁ − p₂)² per group, and when the test will be run with the continuity correction the corrected size is n′ = (n/4)(1 + √(1 + 4/(n |p₁ − p₂|)))²; for one proportion against p₀, n = (z_α √(p₀ q₀) + z_β √(p₁ q₁))² / (p₁ − p₀)² and n′ = n + 1/|p₁ − p₀|. The tool evaluates the Fleiss forms for the alternative proportion that lies h away from the baseline in the observed direction, so the two approaches describe the same alternative. For the chi-square tests the alternative follows the noncentral chi-square with λ = N w², for ANOVA the noncentral F with λ = f² N, and the required N is again the smallest reaching the target (Cohen 1988, ch. 7 and 8; w = 0.3 on 1 degree of freedom needs N = 88, f = 0.25 with four groups 45 per group).

The power reported "at the sample effect" takes the observed effect as the true one. It is a function of the p-value and adds nothing to it: a result with p just above α always shows a power near 0.5, and quoting it as a reason the study "would have found the effect with more data" is circular. Plan the sample size from the smallest effect worth detecting, decided before the data are collected.

Cautions

A p-value is not the probability that H₀ is true, nor 1 − (the probability that the result will replicate); it is a statement about the data given H₀. Statistical significance is not practical importance: report the effect size and the confidence interval with it. When several tests are run on the same data the chance of at least one false positive grows with their number; the Tukey and Bonferroni adjustments in the ANOVA section are the standard remedies, and the same applies to any collection of t tests. The one-sided alternative must be fixed in advance, never chosen after seeing which direction the data went. The t tests assume approximately normal data (or moderate n), independence of the observations and, for the pooled test, equal variances; the proportion and chi-square tests assume independent trials and expected counts of 5 or more; ANOVA assumes equal variances and independent groups. When these fail, a transformation, a Welch-type test or a nonparametric or exact test is the usual recourse.

Worked example: Student's sleep data

Student's 1908 paper tabulated the extra hours of sleep of ten patients under two soporific drugs (R's datasets::sleep). The default case takes the two lines as paired observations: the mean difference is −1.58 h with standard deviation 1.23 h, t = −1.58/(1.23/√10) = −4.06 on 9 degrees of freedom, p = 0.0028 two-sided, 95 % interval −2.46 to −0.70 h, d = −1.28 (large), observed power 0.95. Run as two independent samples the same numbers give Welch t = −1.86 on 17.8 degrees of freedom, p = 0.079: the pairing is what makes the difference detectable. Choosing the two-sample test and summary entry reproduces the Welch and pooled results of R's t.test.