ANOVA and DOE: Method

Analysis of Variance, Multiple Comparisons and Two-Level Factorial Designs

What the analysis of variance does

Fisher's analysis of variance partitions the total variation of a response, SS_T = Σ(y − ȳ)², into the part explained by the factors under study and the part left over as experimental error. Under the hypothesis that a factor has no effect, its mean square and the error mean square both estimate the error variance σ², so their ratio F = MS_factor/MS_E follows the F distribution with the two degrees of freedom; a ratio far above 1 says the factor moved the mean more than chance allows. The p-value is P(F ≥ F₀) under that hypothesis; it is compared with the chosen α (0.05 by default). R² = 1 − SS_E/SS_T is the fraction of the variation the model explains. The analysis assumes independent observations (randomised run order), a constant variance across the treatments and approximately normal errors; the residual plots and the homogeneity tests on the page check the last two.

One-way ANOVA

One factor at a levels (groups) with n_i observations each. The model is y_ij = μ + τ_i + ε_ij with Σ τ_i = 0, and the identity SS_T = SS_Treatments + SS_E with SS_Treatments = Σ n_i(ȳ_i − ȳ)² on a − 1 degrees of freedom and SS_E = Σ(y_ij − ȳ_i)² on N − a. The groups may have different sizes. Each group mean is reported with its confidence interval ȳ_i ± t_{α/2, N−a} √(MS_E/n_i), which uses the pooled error rather than the group's own standard deviation.

Homogeneity of variance

The modified Levene test (Brown and Forsythe 1974) replaces each observation by its absolute deviation from the group median, d_ij = |y_ij − ỹ_i|, and runs the one-way ANOVA on the d_ij; a significant F says the spread differs between groups. Because it works on deviations from medians it is little affected by non-normal tails and is the test to read first. Bartlett's test, χ₀² = 2.3026 q/c with q = (N − a) log₁₀ S_p² − Σ(n_i − 1) log₁₀ s_i² and c = 1 + [Σ 1/(n_i − 1) − 1/(N − a)]/(3(a − 1)), compared with χ² on a − 1 degrees of freedom, is more powerful for normal data but rejects for non-normality as readily as for unequal variances. When the variances do differ, a transformation of the response (log, square root) that stabilises them, or a Welch-type test, is the remedy; the F test itself is fairly robust when the group sizes are equal.

Which means differ: Tukey and Fisher

A significant F says only that the means are not all equal. Tukey's honestly significant difference compares every pair at a family-wise error rate α: two means differ when |ȳ_i − ȳ_j| exceeds T_α = q_α(a, f) √(MS_E/n), where q_α(a, f) is the upper α point of the studentized range of a means with f = N − a error degrees of freedom; the tool evaluates the studentized range distribution by numerical integration rather than from a table, so the value agrees with the tables to the digits they print. For unequal group sizes the Tukey–Kramer margin q_α √(MS_E(1/n_i + 1/n_j)/2) is used. The simultaneous confidence intervals on each difference are the difference ± the margin, and the Tukey p-value of a pair is the probability that the studentized range exceeds the observed |ȳ_i − ȳ_j|/√(MS_E/n). Fisher's least significant difference, LSD = t_α/2,f √(MS_E(1/n_i + 1/n_j)), is the ordinary two-sample t test with the pooled MS_E; it controls the error rate of each comparison, not of the family, so with many groups it declares more differences than it should. Report Tukey when every pair matters; LSD is acceptable after a significant F for a few planned comparisons.

Two-way ANOVA with replication

Two factors, A at a levels and B at b levels, with n replicates in every one of the ab cells (the design must be complete and balanced). The model y_ijk = μ + τ_i + β_j + (τβ)_ij + ε_ijk splits the variation into SS_A = bn Σ(ȳ_i.. − ȳ)², SS_B = an Σ(ȳ_.j. − ȳ)², the interaction SS_AB = n Σ(ȳ_ij. − ȳ_i.. − ȳ_.j. + ȳ)² and the error SS_E = Σ(y_ijk − ȳ_ij.)² on ab(n − 1) degrees of freedom. Replication is what makes the interaction testable: with one observation per cell the interaction and the error cannot be told apart and the tool falls back to the additive model, using the AB sum of squares as the error. The interaction is read first. When it is significant the effect of A depends on the level of B, the main effects alone can mislead, and the comparisons are made between the cell means within each level of the other factor with T = q_α(a, f) √(MS_E/n). When it is not, the marginal means are compared with the standard error of a marginal mean, √(MS_E/(bn)) for A. The main-effect plot draws each factor's marginal means against its levels; the interaction plot draws the cell means against B with one line per level of A: parallel lines mean no interaction, crossing or diverging lines mean a strong one.

2^k factorial designs

A 2^k design runs every combination of k factors at two levels each, coded −1 (low) and +1 (high), and is the workhorse of screening experiments: it estimates every main effect and every interaction from the same runs, each effect using all the data. The effect of a term is the average response at its + runs minus the average at its − runs; computed from the contrast, Effect = Contrast/(n 2^k−1) where the contrast is Σ(±) y with the sign of each run the product of the term's factor columns, and the sum of squares is SS = Contrast²/(n 2^k) on one degree of freedom. The regression coefficient of a term is half its effect, so the fitted model in coded units is ŷ = ȳ + Σ (Effect/2) x_Ax_B… over the terms retained; predictions at the factor levels are read off by substituting ±1 and interpolation between the levels is linear, which is why a 2^k design cannot see curvature (centre points or a 3^k design are needed for that).

With replicates (n > 1) the within-cell variation gives a pure-error estimate on 2^k(n − 1) degrees of freedom and every term is tested by F = SS/MS_E; terms significant at α enter the fitted model. The percent contribution SS/SS_T ranks the terms regardless of significance.

Single replicate: Lenth's method and the half-normal plot

Screening designs are usually run once, which leaves no degrees of freedom for error. Daniel's half-normal plot draws the absolute effects against the half-normal scores Φ⁻¹(½ + ½(i − ½)/m): effects that are only noise fall on a straight line through the origin and the real ones stand off it to the right. Lenth's pseudo standard error makes that judgement numeric: s₀ = 1.5 × median|c_j| is a first robust estimate of the standard error of an effect, the effects beyond 2.5 s₀ are trimmed as probably real, and PSE = 1.5 × median of the rest estimates the standard error from the small effects alone. The margin of error ME = t_{α/2, m/3} PSE tests each effect by itself; the simultaneous margin SME = t_{γ, m/3} PSE with γ = (1 + (1 − α)^1/m)/2 holds the family-wise rate at α, so an effect beyond SME is real beyond doubt and one between ME and SME deserves a look at the plot. The tool keeps the terms beyond ME in the model and pools the rest into an error estimate, which is the usual projection of the design onto its active factors (sparsity of effects); the resulting ANOVA is approximate because the pooled terms were chosen by looking at the data.

Residual checks

Every analysis ends with the residuals e = y − ŷ. Against the fitted values they should scatter in a band of constant width: a funnel opening to the right means the variance grows with the mean and the response should be transformed (log for a constant coefficient of variation, square root for counts); a curve means a missing term. On the normal probability plot the sorted residuals against the normal scores should follow a straight line through the origin of slope s; moderate departures in the tails are of little consequence to the F test, a single far-off point is a candidate outlier to be checked against the run record.

Data formats

• One-way: one group per line, the values separated by spaces, commas or tabs. A label before a colon (160 W: 575 542 …) or a non-numeric first token (A 62 60 63 59) names the group; otherwise the groups are numbered.
• Two-way: one line per cell or per observation, A-level B-level y [y y …]; the levels may be words or numbers (numeric levels are sorted and used as the plot abscissa). Every cell must be present with the same number of replicates.
• 2^k factorial: one line per run, k columns of −1/+1 (or − and +) followed by the response; replicates as extra columns on the line or as repeated runs. All 2^k combinations must be present with equal replication; the number of factors is inferred from the leading coded columns or typed in the Factors field. Factor names are optional.

Worked examples on the page

The default is Montgomery's plasma etching experiment (8th ed., Example 3.1, Table 3.1): four RF power settings, five wafers each. The book prints SS_Treatments = 66,870.55, SS_E = 5,339.20, MS_E = 333.70, F₀ = 66.80, the 95 % interval 533.88 to 568.52 on the 160 W mean (Example 3.3), Bartlett's χ₀² = 0.43 (Example 3.4), Tukey's T_0.05 = 4.05 √(333.70/5) = 33.09 (Example 3.7) and LSD = 24.49 (Example 3.8); the page reproduces them. Choosing the two-way analysis loads the battery-life experiment (Example 5.1: material F = 7.91, temperature F = 28.97, interaction F = 3.56) and the factorial loads the single-replicate filtration-rate 2⁴ (Example 6.2: effects A 21.625, C 9.875, D 14.625, AC −18.125, AD 16.625; Lenth PSE 2.625, ME 6.75, SME 13.70). The test script also checks the pilot-plant 2³ of Box, Hunter and Hunter (Table 5.2), their blood-coagulation one-way example with unequal groups (Table 4.1), Montgomery's replicated 2² (Example 6.1) and the modified Levene example (Example 3.5).

Limits of the tool

• Fixed-effects models only: no random or mixed effects, no blocking, no nesting, no repeated measures.
• Two-way designs must be complete and balanced; unbalanced data need a regression (Type II/III) analysis.
• Factorials are full 2^k designs with k ≤ 5: no fractional factorials, centre points, blocking or three-level factors, and no response-surface models.
• No Welch or Kruskal–Wallis alternatives when the variances differ; transform the response instead.
• The pooled-error ANOVA of a single-replicate factorial is conditional on the effects Lenth's margin selected and is approximate.