Diagnostic Accuracy and Agreement

Diagnostic Accuracy and Agreement

Diagnostic Accuracy and Agreement

Two different questions

A study of a test, a rater or a measuring instrument asks one of two questions. Accuracy: how often is the result right? That needs a reference standard, the best available way of establishing whether the condition is present. Agreement: how closely do two results match, when neither is known to be right? Two tests can agree and both be wrong, so agreement is not accuracy, and the FDA guidance of 2007 asks that results against a non-reference standard be reported as percent agreement and never as sensitivity or specificity. The first five analyses of this page are about accuracy, the last four about agreement.

The 2 × 2 table

With a reference standard the subjects fall into four cells: true positives TP, false positives FP, false negatives FN and true negatives TN. Sensitivity = TP / (TP + FN) is the share of subjects with the condition whom the test finds; specificity = TN / (FP + TN) the share without it whom the test clears. Both are properties of the test in the kind of subjects studied and do not depend on how many of each were sampled. The positive predictive value TP / (TP + FP) and the negative predictive value TN / (TN + FN) answer the clinical question, what a positive or a negative result means, but they depend on the prevalence p of the condition. By Bayes' theorem

PPV = Se p / [Se p + (1 − Sp)(1 − p)],   NPV = Sp (1 − p) / [Sp (1 − p) + (1 − Se) p].

The predictive values counted in the sample hold only at the sample's prevalence, and mean nothing in a case-control sample where the numbers with and without the condition were fixed by the investigator; enter the prevalence of the population in which the test will be used to get the values that apply there.

The likelihood ratios LR+ = Se / (1 − Sp) and LR− = (1 − Se) / Sp say how much a result shifts the odds of the condition: post-test odds = pre-test odds × LR. The diagnostic odds ratio (TP × TN) / (FP × FN) = LR+ / LR− condenses the table into one number, and the Youden index J = Se + Sp − 1 runs from 0 (useless) to 1 (perfect). Accuracy (TP + TN) / n is given for completeness; like the predictive values it depends on prevalence.

Confidence limits of a proportion

Each proportion x / n carries two intervals. The Wilson score interval, the one printed in the FDA guidance, is

[p̂ + z² / (2n) ± z √(p̂ (1 − p̂) / n + z² / (4n²))] / (1 + z² / n),

with z the normal quantile of the confidence level. The exact (Clopper–Pearson) interval inverts the binomial distribution: its limits are the quantiles α/2 of Beta(x, n − x + 1) and 1 − α/2 of Beta(x + 1, n − x). It never covers less than the stated level and is therefore somewhat wider. Neither needs the large counts that the simple p̂ ± z √(p̂ q̂ / n) interval needs, which fails for proportions near 0 or 1, exactly where good tests live.

The limits of a likelihood ratio are found on the log scale: ln LR+ ± z √[1/TP − 1/(TP + FN) + 1/FP − 1/(FP + TN)], and the same with FN and TN for LR−. The odds ratio uses ln DOR ± z √(1/TP + 1/FP + 1/FN + 1/TN), with ½ added to every cell when one is zero. The limits of a predictive value at an entered prevalence come from carrying the likelihood-ratio limits through Bayes' theorem, which is the logit method of Mercaldo, Lau and Zhou. The Youden index has a normal-approximation interval and a Newcombe interval built from the Wilson limits of sensitivity and of the false-positive rate; the second behaves better near 1.

Comparing two tests

Same subjects. When both tests are read on every subject, only the discordant subjects carry information about which test is more sensitive: b subjects positive on test 1 alone and c on test 2 alone, among those with the condition. Under equal sensitivity they should split evenly, so McNemar's exact test is the two-sided binomial probability of a split at least as uneven as b : c with chance ½. The exact test is conservative; the mid-p value, which counts only half the probability of the observed split, is closer to the nominal level and is the one recommended by Fagerland, Lydersen and Laake. The large-sample statistic (b − c)² / (b + c) is given with and without the continuity correction. The difference of the two sensitivities is (b − c) / n with standard error √[b + c − (b − c)² / n] / n. Specificity is compared in the same way among the subjects without the condition.

Separate samples. When each test has its own subjects, the two sensitivities are independent proportions: the z test with the pooled proportion, Fisher's exact test and the Newcombe interval of the difference, which combines the two Wilson intervals as d − √[(p̂₁ − l₁)² + (u₂ − p̂₂)²] to d + √[(u₁ − p̂₁)² + (p̂₂ − l₂)²].

Sample size

A diagnostic accuracy study is usually sized so that sensitivity and specificity are estimated to a wanted precision. For a proportion expected at P and a half-width d the normal approximation gives n = z² P (1 − P) / d²; the page also solves for the n at which the Wilson interval has that half-width, which is a little larger for proportions near 1. The n for sensitivity counts subjects with the condition and the n for specificity subjects without it, so with prevalence p among those recruited the totals are n / p and n / (1 − p) (Buderer). The larger governs.

ROC analysis

A test that returns a score becomes a positive or negative result only once a cut-off is chosen, and every cut-off trades sensitivity against specificity. The receiver operating characteristic curve plots sensitivity against 1 − specificity for every possible cut-off; the page uses the cut-offs midway between consecutive distinct scores. The area under the empirical curve, by the trapezoid rule, equals the Mann–Whitney statistic: the probability that a randomly chosen subject with the condition scores higher than a randomly chosen subject without it, ties counting ½. It is 0.5 for a useless score and 1 for a perfect one.

Two standard errors are given. Hanley and McNeil's is √{[A (1 − A) + (m − 1)(Q₁ − A²) + (n − 1)(Q₂ − A²)] / (m n)} with Q₁ = A / (2 − A) and Q₂ = 2A² / (1 + A), for m subjects with and n without the condition; it rests on an assumed shape of the score distributions. DeLong's is nonparametric. For each subject with the condition let V₁₀ be the share of subjects without it that score lower (ties ½), and for each subject without it let V₀₁ be the share of subjects with the condition that score higher. The area is the mean of either set, and its variance is var(V₁₀) / m + var(V₀₁) / n. When two scores are measured on the same subjects the covariance of their areas is cov(V₁₀⁽¹⁾, V₁₀⁽²⁾) / m + cov(V₀₁⁽¹⁾, V₀₁⁽²⁾) / n, and the two areas are compared by z = (A₁ − A₂) / √[var(A₁) + var(A₂) − 2 cov(A₁, A₂)].

Cut-offs. The Youden cut-off maximizes J = sensitivity + specificity − 1, the height of the curve above the chance line. The closest-to-the-corner cut-off minimizes (1 − Se)² + (1 − Sp)². Both treat a false negative and a false positive as equally bad and ignore prevalence. The cost-weighted cut-off maximizes Se + r Sp with r = (1 − p) / (c p), where c is the cost of a false negative relative to a false positive and p the prevalence: a costly false negative or a common condition lowers r and moves the cut-off toward sensitivity. A cut-off chosen and judged on the same data is optimistic and should be confirmed on new subjects.

Partial area. When only high specificities are of practical use (screening), the area can be restricted to a specificity range. The standardized form of McClish, ½ [1 + (area − chance) / (largest − chance)], puts it back on the 0.5 to 1 scale.

Agreement of raters: Cohen's kappa

Two raters who each put n subjects into k categories agree on the share p_o on the diagonal of their cross-table. Some of that would happen by chance: with row shares p_i· and column shares p_·i the expected agreement is p_e = Σ p_i· p_·i. Cohen's coefficient κ = (p_o − p_e) / (1 − p_e) is the share of the possible agreement beyond chance that was achieved: 1 for perfect agreement, 0 for chance, negative for less than chance.

On an ordered scale a near miss is better than a wide one. Weighted kappa replaces the diagonal count by Σ w_ij p_ij and the chance value by Σ w_ij p_i· p_·j, with weights w_ij = 1 − |i − j| / (k − 1) (linear) or 1 − (i − j)² / (k − 1)² (quadratic). Quadratic weighted kappa is close to an intraclass correlation of the category numbers. The standard errors are the large-sample values of Fleiss, Cohen and Everitt: one for the confidence interval and another, valid under chance agreement, for the test of κ = 0.

Kappa depends on the category totals. In a 2 × 2 table with cells a, b, c, d the prevalence index (a − d) / n measures how unequal the two categories are and the bias index (b − c) / n how differently the raters use them; a large prevalence index lowers κ for the same percent agreement. The prevalence-adjusted bias-adjusted value PABAK = 2 p_o − 1 (Byrt, Bishop and Carlin) is reported beside κ, with the agreement on each category, 2a / (2a + b + c) and 2d / (2d + b + c), and the FDA's positive and negative percent agreement a / (a + c) and d / (b + d). The Landis–Koch labels (slight to 0.20, fair to 0.40, moderate to 0.60, substantial to 0.80, almost perfect above) are a convention.

Several raters: Fleiss' kappa

When each of N subjects is rated by n raters, not necessarily the same ones, let n_ij be the number who put subject i in category j. The share of agreeing rater pairs on subject i is P_i = (Σ_j n_ij² − n) / [n (n − 1)], and P̄ is its mean. With p_j the share of all ratings in category j, chance agreement is P̄_e = Σ p_j² and κ = (P̄ − P̄_e) / (1 − P̄_e). Its standard error under chance agreement is √{2 [S² − Σ p_j q_j (q_j − p_j)] / [S² N n (n − 1)]} with q = 1 − p and S = Σ p_j q_j. A coefficient is also given for each category against all others, with standard error √[2 / (N n (n − 1))].

Agreement of measurements: Bland and Altman

Two methods that measure the same quantity are compared through the differences d = A − B on the same subjects. The mean difference d̄ is the bias of one method relative to the other and, if the differences are roughly normal and unrelated to the size of the measurement, 95 % of them lie between the limits of agreement d̄ ± 1.96 s, where s is their standard deviation (Bland and Altman used 2 s). Whether the methods can replace each other is decided by comparing those limits with the largest difference that would not matter in practice; a correlation coefficient cannot decide it, because it measures association along any line and grows with the range of the sample. The standard error of d̄ is s / √n and that of a limit about s √[(1 + k² / 2) / n] for limits at k standard deviations, which is √(3 s² / n) for k = 2; the intervals use Student's t with n − 1 degrees of freedom.

The plot of difference against mean shows whether the assumptions hold. A slope in it means proportional bias; the page fits the least-squares line of difference on mean and tests its slope. When the scatter grows with the mean, a log transformation is the usual remedy.

Repeatability. With a repeat measurement by each method, the standard deviation of the repeat differences (about zero) is s = √(Σ d² / n) and the repeatability coefficient k s is the difference two measurements by the same method seldom exceed. Poor repeatability limits the agreement any other method can have with it. When the methods are compared through the means of their two measurements, the standard deviation of the differences is corrected to s_c = √(s_D² + ¼ s_A² + ¼ s_B²).

Lin's concordance correlation ρ_c = 2 s_AB / [s_A² + s_B² + (Ā − B̄)²] measures how closely the points lie to the line of equality. It is the Pearson correlation r multiplied by a bias factor C_b ≤ 1 that falls with a shift in location or scale. Its interval is found on Fisher's z scale.

Deming regression fits B = a + b A allowing measurement error in both methods, with δ the ratio of the error variance of B to that of A: b = {s_BB − δ s_AA + √[(s_BB − δ s_AA)² + 4 δ s_AB²]} / (2 s_AB). With δ = 1 it is orthogonal regression. Standard errors are by the jackknife (Linnet). An intercept interval excluding 0 points to a constant difference, a slope interval excluding 1 to a proportional one. Passing–Bablok regression is not computed.

Intraclass correlation

For n subjects each rated by k raters, a two-way analysis of variance splits the variation into mean squares between subjects (BMS), between raters (JMS) and residual (EMS); the within-subject mean square WMS pools the last two. Shrout and Fleiss define three single-rating coefficients by the design:

ICC(1,1) = (BMS − WMS) / [BMS + (k − 1) WMS], each subject rated by a different set of raters;
ICC(2,1) = (BMS − EMS) / [BMS + (k − 1) EMS + k (JMS − EMS) / n], the same raters, regarded as a sample of all raters, absolute agreement;
ICC(3,1) = (BMS − EMS) / [BMS + (k − 1) EMS], the same raters and no others, consistency.

The corresponding coefficients for the mean of the k ratings are (BMS − WMS) / BMS, (BMS − EMS) / [BMS + (JMS − EMS) / n] and (BMS − EMS) / BMS. The confidence limits come from the F distributions of the mean-square ratios; for ICC(2,1) they use Satterthwaite's approximate degrees of freedom, as given by Shrout and Fleiss. A systematic difference between raters lowers ICC(2,1) but not ICC(3,1), so the choice between them is the choice between asking for the same number and asking for the same ordering.

What the page does not do

Smoothed or binormal ROC curves, bootstrap intervals, confidence intervals of the partial area and of the cut-off, and tests of more than two curves; Passing–Bablok regression; Bland–Altman limits for repeated measures beyond one repeat per method; kappa for raters who rate different subjects with unequal numbers of ratings; Light's and Conger's coefficients; Krippendorff's alpha; ICC with missing ratings; verification-bias or imperfect-reference corrections. The Newcombe interval is used for independent proportions; the paired difference has the simple large-sample interval only.