Survey Analysis: Method

Proportions, Margins of Error, Cross-Tabs, Likert Items and NPS

What the tool takes

Survey responses in two layouts. Respondent rows: a first line of column names, then one line per respondent with the cells separated by commas, tabs or semicolons (whitespace when none of these is present). Two column names are reserved: count repeats a line that many times, so a table of identical respondents can be typed as one line per pattern (the default data are typed that way), and weight carries each respondent's survey weight (a post-stratification or design weight). Both may be present. A cross-tab of counts is the other layout: a header line of column categories (with or without a corner label), then one line per row category followed by its counts; the two names typed in the Cross-Tab group become the row and column variables and the tool treats the table as counted rows with two columns. Blank cells and the token NA are missing values and are excluded from the table that reads them.

Weights and the design effect

With weights w_i a proportion is estimated as p̂ = Σ w_i I(x_i = k) / Σ w_i. Unequal weights make the estimate less precise than an equal-weight sample of the same size; Kish's design effect for this loss is deff = n Σ w_i² / (Σ w_i)² = 1 + CV²(w), the squared coefficient of variation of the weights plus one (Kish 1965 §8.2; Kish 1992). The effective sample size n_eff = n / deff is the number of equal-weight respondents that would give the same variance, and it is what the intervals, the margins of error and the NPS standard error use. This is the design effect of the weighting alone: clustering and stratification in the sampling plan add their own effects, which the tool does not know about and the analyst must supply by inflating the weights' deff or by reading the intervals as optimistic.

Frequency tables and the Wilson interval

Each chosen column is tabulated: count, proportion (weighted when weights are present), the Wilson score interval and the margin of error. The Wilson (1927) interval is the set of p for which the observed proportion is not significantly different from p at level α: (p̂ + z²/2n ± z √(p̂(1 − p̂)/n + z²/4n²)) / (1 + z²/n), with n = n_eff. Newcombe (1998) compared seven methods and recommended it for routine use: unlike the Wald interval p̂ ± z √(p̂(1 − p̂)/n) it never leaves [0, 1], behaves well for small counts and gives a non-degenerate interval when the count is 0 or n (0 of 20 gives 0 to 16.1 %). The margin of error column is the Wald half-width z √(p̂(1 − p̂) deff / n) times the finite-population correction, the figure a report quotes as "± x points"; the two agree closely when the proportion is not near 0 or 1 and n is in the hundreds.

Margin of error and sample size

The overall margin of error of the survey is the Wald half-width at a planning proportion p₀: MOE = z √(p₀(1 − p₀) deff / n) × √((N − n)/(N − 1)). With p₀ = 0.5 it is the largest margin any proportion in the survey can have (the usual "accurate to ± 3 points" statement); the finite-population correction applies when the sample is a noticeable fraction of a population of known size N. For planning a new survey Cochran (1977 §4.4) gives n₀ = z² p₀(1 − p₀) / E² for a target margin E and, for a finite population, n = n₀ / (1 + (n₀ − 1)/N). At 95 % confidence and E = 5 points, n₀ = 384.2; this is the formula behind the Krejcie and Morgan (1970) table (which uses χ²₁ = 3.841 for z² and rounds to the nearest: 384 at N = 1,000,000, 278 at N = 1,000, 80 at N = 100). The page prints the exact value and the integer rounded up. With a design effect, multiply the required n by deff.

Cross-tabs and the test of independence

The cross-tab of two columns shows each cell's count with its row percentage (the distribution of the column variable within each row category, the usual way to read a cross-tab), its column percentage and its expected count under independence μ_ij = n_i+ n_+j / n. Pearson's statistic X² = Σ (n_ij − μ_ij)² / μ_ij and the likelihood-ratio statistic G² = 2 Σ n_ij ln(n_ij / μ_ij) both follow the χ² distribution with (r − 1)(c − 1) degrees of freedom for large samples (Agresti 2002 §3.2); the two agree closely when the expected counts are above about 5, and the page flags cells below that. Cramér's V = √(X² / (n (min(r, c) − 1))) puts the strength of the association on a 0 to 1 scale, independent of n; values below 0.1 are negligible, 0.1 to 0.3 weak, 0.3 to 0.5 moderate. The adjusted standardized residuals (n_ij − μ_ij) / √(μ_ij(1 − p_i+)(1 − p_+j)) (Haberman 1973; Agresti 2002 §3.3.1) are approximately standard normal under independence and show which cells carry the association: in the default table the residuals of ± 2.29 for Democrat and ± 2.62 for Republican, against 0.46 for Independent, locate the gender difference in the two major parties.

For a 2 × 2 table the page adds the φ coefficient (ad − bc) / √((a + b)(c + d)(a + c)(b + d)), which equals V with the sign of the association, the sample odds ratio ad / (bc), the Yates continuity-corrected statistic Σ (|n_ij − μ_ij| − ½)² / μ_ij, and Fisher's exact test (Fisher 1935; Agresti 2007 §2.6): with both margins taken as fixed the count in cell (1,1) is hypergeometric, the one-sided p-value sums the probabilities of counts at least as extreme in the observed direction, and the two-sided p-value sums the probabilities of every table as or less probable than the observed one (the definition R uses). Fisher's tea-tasting table [[3, 1], [1, 3]] gives 17/70 = 0.243 and 34/70 = 0.486.

With survey weights the table of weighted counts, scaled to n, gives the population percentages, and X² computed on it is divided by the design effect before it is referred to χ²: the first-order Rao and Scott (1984) correction, which treats the weighting deff as the mean generalized design effect. It is approximate; a full design-based test needs the replicate weights or the strata and clusters of the sampling plan.

Likert items

A Likert item (Likert 1932) is an ordered rating, here 1 to 5 or 1 to 7. The page reports the mean and standard deviation, which read the points as equally spaced (the interval treatment defended by Carifio and Perla 2007 for summated scales, and the usual practice for single items when the distribution is not pathological), the standard error of the mean, the top-two-box share (the two highest points, "agree" and "strongly agree" on a 5-point agreement scale), the bottom-two-box share (points 1 and 2), the net score top-two-box minus bottom-two-box in percentage points, and the full distribution over the points. Non-integer values and values outside the scale are listed as invalid and excluded. The distribution table and chart are the honest summary of an item: two items with the same mean can have very different polarisation, and the top-two-box figure is what most survey reports quote.

Net Promoter Score

The 0–10 "how likely are you to recommend" item (Reichheld 2003) is cut into promoters (9–10), passives (7–8) and detractors (0–6), and NPS = 100 (p_P − p_D), between −100 and +100. Because the two shares come from the same multinomial sample, Var(p_P − p_D) = (p_P + p_D − (p_P − p_D)²) / n (Rocks 2016, eq. 2), so SE(NPS) = 100 √Var, with n_eff in place of n under weighting, and the confidence interval is NPS ± z SE. Rocks compared this Wald interval with several alternatives and found it adequate for n above about 100; for small samples the interval is reported but should be read as approximate. The sign of a score is established only when its interval excludes zero, and two surveys' scores differ significantly only when the difference exceeds about z times the root sum of squares of their standard errors. The page caps the interval at ± 100.

Worked example on the page

The default data are the gender × party identification table of the 1991 General Social Survey from Agresti (2002, Table 3.2): females 279 Democrat, 73 Independent, 225 Republican; males 165, 47, 191; n = 980. The book prints X² = 7.01, G² = 7.00, df = 2, P = 0.03 and the standardized residuals ± 2.29, ± 0.46, ± 2.62 (Table 3.3); the page reproduces them from the rows typed with a count column. The satisfaction (1–5) and recommend (0–10) columns are illustrative values attached to those rows to show the Likert and NPS sections; they are not survey data. The test script checks the library against these values, the 2000 GSS table of Agresti (2007, Table 2.5: X² = G² = 30.0, residuals ± 4.50, ± 0.70, ± 5.32), Fisher's tea-tasting p-values, Newcombe's Wilson intervals, the Krejcie–Morgan table, Kish's design effect and a numpy Monte Carlo of the NPS standard error.

Limits of the tool

• Simple random sampling with weights: clustering and stratification enter only through the weights' design effect. Replicate weights, strata and PSUs are not read.
• Intervals are the Wilson score and Wald forms; no exact (Clopper–Pearson) interval, no Bayesian interval.
• The cross-tab tests one pair of columns; multi-way tables, log-linear models and ordinal measures (gamma, Kendall's tau-b) are not offered.
• Fisher's exact test is for 2 × 2 tables only; larger sparse tables need an exact network algorithm or a Monte Carlo test.
• Likert items are summarised one at a time; reliability (Cronbach's α) and summated scales are not computed.
• Up to eight frequency tables and eight Likert items per run; the data box is read in full, so very large files are slow.