Data Summary and Outliers

Data Summary and Outliers

Descriptive Statistics, Normality Tests and Outlier Tests

What the tool does

A pasted table (one row per line, cells separated by commas, tabs, semicolons or spaces, the first line read as the header when it holds any non-numeric cell) is summarized column by column. Missing cells (blank, NA, NaN, a dot or a dash) are counted and skipped. For every chosen column the tool reports the location, spread and shape measures of the summary table, three tests of normality, four outlier procedures, and, when two or more columns are chosen, their pairwise correlations on the rows both share. One column, chosen by number, gets the worked equations with the substituted numbers and the three figures. The default data are the NIST/SEMATECH e-Handbook heat flow meter calibration set (case study 1.4.2.8, 195 values), whose printed summary statistics, confidence intervals, lag-1 autocorrelation and Grubbs statistic the page reproduces, beside an illustrative 12-value column with one planted outlier that shows every outlier test firing.

Location and spread

The mean and the standard deviation s (denominator n − 1) describe a roughly symmetric sample; the standard error s/√n is the standard deviation of the mean itself, and the 95 % confidence interval of the mean is mean ± t_{0.975, n−1} s/√n with Student's t. The interval of s uses the chi-square distribution of (n − 1)s²/σ². The median, the quartiles and the interquartile range IQR = Q₃ − Q₁ describe the sample without being pulled by its extremes. Quartiles have several definitions, and two are shown: the type 7 rule of Hyndman and Fan (1996), linear interpolation of the order statistics at position (n − 1)p + 1, which R, NumPy, Excel and most spreadsheets use by default; and Tukey's hinges (1977), the medians of the lower and upper halves of the ordered sample, the sample median counted in both halves when n is odd (the depth of a hinge is (⌊(n + 1)/2⌋ + 1)/2, a fractional depth averaging the two values about it). The two agree for many n and differ a little for others; the box plot and the fences follow Tukey. The coefficient of variation is s/|mean|. The median absolute deviation MAD = median|x_i − median| is a robust scale; multiplied by 1.4826 = 1/Φ⁻¹(0.75) it estimates σ for a normal sample. The 10 % trimmed mean drops the ⌊0.1n⌋ smallest and largest values and averages the rest, a compromise between the mean and the median.

Shape: skewness and kurtosis

With the central moments m_k = Σ(x_i − x̄)^k/n, the Fisher–Pearson skewness is g₁ = m₃/m₂^3/2 and the excess kurtosis g₂ = m₄/m₂² − 3, both zero for a normal distribution (NIST 1.3.5.11). Joanes and Gill (1998) compare the sample versions; the adjusted forms G₁ = g₁ √(n(n − 1))/(n − 2) and G₂ = ((n − 1)/((n − 2)(n − 3))) ((n + 1) g₂ + 6) are what Excel, SAS, SPSS and Minitab print and are less biased in small samples. Under normality the standard errors are approximately √(6/n) for the skewness and √(24/n) for the kurtosis; the exact small-sample values are SES = √(6n(n − 1)/((n − 2)(n + 1)(n + 3))) and SEK = 2 SES √((n² − 1)/((n − 3)(n + 5))). A ratio G/SE beyond about ±2 marks a departure worth noting; the formal tests below put a p-value on it. Positive skewness is a longer right tail; positive excess kurtosis means tails heavier than normal (leptokurtic), typically with a sharper peak, and negative excess kurtosis lighter tails (platykurtic), as in a uniform sample.

Normality tests

All three tests take the mean and variance as unknown and estimated from the sample (the "case 3" of the goodness-of-fit literature), which changes the null distribution of the statistics; the critical values and p-values used here are the ones for that case.

• Shapiro–Wilk W = (Σ a_i x_(i))² / Σ(x_i − x̄)², the squared correlation between the ordered sample and the expected normal order statistics, with the coefficients a_i and the normalizing transform of Royston's (1995) Algorithm AS R94 (3 ≤ n ≤ 5000; the same code as R's shapiro.test and SciPy). W is at most 1; small W rejects normality. It is the most powerful general-purpose test for small and moderate samples.
• Anderson–Darling A² = −n − (1/n) Σ (2i − 1)[ln F(x_(i)) + ln(1 − F(x_(n+1−i)))] with F the normal cdf at the sample mean and s weights the tails of the empirical distribution more than the Kolmogorov–Smirnov statistic (NIST 1.3.5.14). For estimated parameters D'Agostino and Stephens (1986, Table 4.7) give the modified statistic A* = A²(1 + 0.75/n + 2.25/n²) with the 5 % critical value 0.752 and, in Table 4.9, a p-value formula in A*; both are reported. Stephens' earlier (1974) modification A²(1 + 4/n − 25/n²) with the critical value 0.787 is what Dataplot and the NIST case studies print as the "adjusted test statistic", so it is shown too: the two forms reach the same decision.
• D'Agostino–Pearson K² (D'Agostino, Belanger and D'Agostino 1990) tests the skewness and the kurtosis together. The skewness g₁ is transformed by D'Agostino's (1970) formula into Z₁ (n ≥ 8) and the kurtosis b₂ = g₂ + 3 by the Anscombe–Glynn (1983) formula into Z₂, each approximately standard normal under normality; K² = Z₁² + Z₂² is approximately χ² with 2 degrees of freedom, so p = exp(−K²/2) and the 5 % critical value is 5.991. The two Z values say how normality fails (asymmetry or tail weight); the kurtosis transform is rough below about n = 20. These are the formulas of SciPy's skewtest, kurtosistest and normaltest.

The normal probability plot correlation coefficient PPCC (Filliben 1975; NIST 1.3.3.21) is the correlation of the ordered sample with the normal quantiles of the uniform order-statistic medians; it is the straightness of the Q–Q plot as one number, 1 for a perfectly normal sample. The lag-1 autocorrelation r₁ = Σ(x_i − x̄)(x_i+1 − x̄)/Σ(x_i − x̄)² of the column in its pasted order is reported because every standard error above assumes independent values; |r₁| beyond about 2/√n says consecutive values are related and the uncertainty of the mean is larger than s/√n (the heat flow meter data show r₁ = 0.28, which NIST notes as a mild violation).

Outlier procedures

An outlier is an observation that deviates markedly from the rest. Finding one is a reason to investigate, not to delete: a recording or transcription error is corrected or removed, a genuine extreme is kept (it may be the most informative value in the set) or handled with the robust measures above. All the formal tests assume the bulk of the data is normal, so a rejection of normality and an outlier can be the same fact seen twice; NIST 1.3.5.17 recommends looking at the normal probability plot first. Four procedures are run:

• Tukey's fences (1977): with the hinges H_L, H_U and the H-spread (the hinges' IQR), the inner fences are H_L − 1.5 IQR and H_U + 1.5 IQR and the outer fences H_L − 3 IQR and H_U + 3 IQR. Values between the inner and outer fences are "outside", beyond the outer fences "far out". The rule is descriptive, needs no distribution and no significance level, and is what a box plot shows; for a normal sample about 0.7 % of values fall outside the inner fences by chance, so in a large sample a few outside values are expected.
• Grubbs' test (1969; NIST 1.3.5.17.1) tests one value, the most extreme, with G = max|x_i − x̄|/s. The two-sided critical value is G_crit = ((n − 1)/√n) √(t²/(n − 2 + t²)) with t the upper α/(2n) quantile of Student's t with n − 2 degrees of freedom (α/n for a one-sided test of a named end), and the p-value inverts that relation. The tool repeats the test on the remaining values while it rejects. That sequential use is common but can fail by masking: two outliers inflate s so much that neither is found at the first step, and the procedure stops with none, as it does on Rosner's 54-value example. The generalized ESD test exists for that case.
• Dixon's ratio (1951) uses gaps between ordered values instead of the mean and s, so it is not inflated by the suspect value itself, and is meant for small samples (n ≤ 30). Rorabacher (1991) recommends r₁₀ = (x₍₂₎ − x₍₁₎)/(x_(n) − x₍₁₎) for n = 3–7, r₁₁ = (x₍₂₎ − x₍₁₎)/(x_(n−1) − x₍₁₎) for 8–10, r₂₁ = (x₍₃₎ − x₍₁₎)/(x_(n−1) − x₍₁₎) for 11–13 and r₂₂ = (x₍₃₎ − x₍₁₎)/(x_(n−2) − x₍₁₎) for 14–30, each mirrored for the high end; the higher ratios protect the test against a second outlier at either end. The suspect is the end with the larger ratio, compared with Rorabacher's two-tailed critical values at 90, 95 or 99 % (the level chosen on the form). For n ≤ 10 the simpler Q = gap/range of Dean and Dixon (1951), the "Q test" of analytical chemistry, is also shown with its own critical value. Dixon's test finds at most one outlier.
• Generalized ESD test (extreme studentized deviate, Rosner 1983; NIST 1.3.5.17.3) for up to k_max outliers: at step i the most extreme remaining value gives R_i = max|x − x̄|/s computed on the values left, and is removed; the critical value is λ_i = (n − i) t_{p, n−i−1}/√((n − i − 1 + t²)(n − i + 1)) with p = 1 − α/(2(n − i + 1)). The number of outliers is the largest i for which R_i > λ_i, not the first: that is what defeats masking. The test needs only an upper bound on the number of outliers, holds its overall level α approximately for n ≥ 25 and is the recommended procedure when more than one outlier is possible. k_max is capped at n − 3.

The outlier table lists every value any procedure flags, with its row in the pasted table and the procedures that flagged it, so each can be traced to its record.

Correlation

For every pair of chosen columns, on the rows where both have a number, Pearson's r = Σ(x − x̄)(y − ȳ)/√(Σ(x − x̄)² Σ(y − ȳ)²) measures linear association. Under the hypothesis of no correlation in a bivariate normal population t = r √((n − 2)/(1 − r²)) follows Student's distribution with n − 2 degrees of freedom, which gives the two-sided p-value. Spearman's ρ is Pearson's r of the ranks, ties given their average rank, and measures any monotone association; the same t approximation gives its p (as SciPy does). The two together are diagnostic: Pearson high and Spearman low means a few extreme rows drive the linear fit; Spearman high and Pearson lower means a monotone but curved relation. Anscombe's (1973) four data sets, all with r = 0.816 and the same regression line but entirely different scatter plots, are the standard warning that a coefficient is not a substitute for looking at the data.

Figures

The histogram (Figure 1) shows the values as a probability density with the normal curve of the sample mean and s, in max(6, min(25, ⌈√n⌉)) equal bins. The box plot (Figure 2) is Tukey's: the box spans the hinges, the line is the median, the whiskers end at the adjacent values (the most extreme data inside the inner fences), the dashed lines are the inner fences, and the points beyond are the outside and far-out values. The normal Q–Q plot (Figure 3) plots the ordered values against the normal quantiles of the Blom plotting positions (i − 3/8)/(n + 1/4) with the line x̄ + s·z: a straight line is normal, a bow is skew, an S is heavy or light tails, and points departing at an end are the outliers.

Reading the results

Start with the figures and the shape measures, then the normality table, then the outliers. If the normality tests reject and the outlier table names one or two values, remove them (after checking the records) and read the tests again: a normal body with a bad value is a different finding from a skewed population. If the lag-1 autocorrelation is large, the data are a time series and the confidence interval of the mean is too narrow; a run-sequence or control chart is the right next tool. The summary table is laid out for a report, statistics down and columns across; the quartile definition should be named when the numbers are quoted, since spreadsheets and texts differ.

Limits

Numbers only (dates and text columns are counted as missing); one table at a time; Shapiro–Wilk needs 3 ≤ n ≤ 5000, Anderson–Darling n ≥ 5, the skewness transform n ≥ 8 and the kurtosis transform is approximate below n ≈ 20; Dixon's table ends at n = 30; the Grubbs, Dixon and ESD tests assume an underlying normal distribution, so on a skewed column they flag the long tail; Tukey's fences on a skewed column flag the long tail too. Not included: grouped or weighted data, the Tietjen–Moore test (which needs the exact number of outliers), robust estimators beyond MAD and the trimmed mean, multivariate outliers, transformations (Box–Cox) and confidence intervals of the median and quartiles.