Sensitivity Analysis

Sensitivity Analysis

Tornado, Spider, Elasticities, Scenarios, Break-Even and Sobol' Indices

What sensitivity analysis is for

A model of a decision, a cost, a design margin or a forecast has inputs that are not known exactly. Sensitivity analysis asks which of them matter: which inputs move the result enough to change the decision, and which can be fixed at a best guess without loss. Clemen and Reilly make it the test of a requisite model, one whose form and detail are sufficient for the decision at hand: the variables that the tornado shows to be unimportant are dropped, the few that dominate are studied further, modelled probabilistically or negotiated. The tool takes a model typed as a formula over named variables, each with a base (best-guess) value and a low and a high value, and runs the deterministic methods on that range; when the variables also carry probability distributions it runs the global, variance-based methods.

The model and the variable table

The model is one or more formulas separated by semicolons, name = expression, in the variable names; a later formula may use an earlier name as an intermediate quantity (Revenue and Cost before Profit), and the last formula is the output that is analyzed. The syntax is plain: + − * / ^, parentheses, sqrt, exp, ln, sin, cos, tan, abs, min, max, the constants pi and e. The tool tracks no units: every variable is entered in its own unit and the output carries the unit the formula gives. Each variable line is name, base, low, high; a line with a base value alone is a fixed quantity (the number of seats, the project life) that is listed but not swept. The ranges are the analyst's judgement of what is plausible: a 10–90 % credible range is the usual choice, and the result of every method below depends on them, so they are worth as much care as the base values.

One-way sensitivity and the tornado diagram

One-at-a-time analysis holds every variable at its base value except one, which is set to its low and then to its high value; the two outputs give that variable's swing. The tornado diagram draws the swings as horizontal bars about the base output, the widest at the top, so the shape names the method: the wide bars at the top are the variables that deserve attention, the narrow ones at the foot can be fixed. The bar of a variable whose increase lowers the output runs the other way, and the tool colours each end by whether the variable sits at its low or its high value. The table also gives the swing as a percentage of the base output and its share of the sum of squared swings, a rough apportioning of the one-way variation. Two cautions: the swing depends on the width of the range the analyst gave, so a variable with a timid range looks unimportant; and the method cannot see interactions, the effect of two variables moving together, which the scenario table and the global indices do.

Spider plot

The spider plot (Eschenbach 1992) sweeps each variable across its range in steps and plots the output against the position in the range, every variable on the same axes. Here the horizontal axis is 100 (x − base)/(high − low), so each variable's base sits at zero and its ends lie at the two sides, which keeps variables of any magnitude and any asymmetry of range on one chart; a steep line is a sensitive variable and a curved one a nonlinear response, which the two-point swing of the tornado would miss (a variable whose line rises and falls inside the range has a small end-to-end swing and a large effect). The tool flags such non-monotone variables.

Elasticities at the base case

The local sensitivity is the derivative ∂y/∂x_i at the base case, computed by central differences at two step sizes (1 % and 0.5 % of x_i) combined by Richardson extrapolation, which is exact for polynomials up to the fourth degree. The elasticity e_i = (∂y/∂x_i)(x_i/y) is the percentage change of the output per percent change of the input, a dimensionless number that compares variables of different units, and the semi-elasticity (∂y/∂x_i) x_i is the change of the output for a 100 % change of the input. The elasticity ranking is local (it ignores the width of the ranges) and undefined where the output or the input is zero; it is the right measure when every input is uncertain by about the same relative amount, and the tornado is the right one when the ranges differ.

Scenarios

The scenario table shows the output, and every intermediate quantity, for the base case; the best and worst cases, in which every variable sits at its favourable or unfavourable end (the end is read from the direction of its one-way swing, and a variable with no effect stays at base; the Goal setting says whether a higher or a lower output is better); the all-low and all-high cases, which set every variable to the same end regardless of direction (the first-cut comparison of Clemen and Reilly, who use it to ask whether it is a best–worst analysis: it is not, because a low operating cost is good news); and the analyst's own scenarios, typed as name: var = value, var = value, with unlisted variables at base (low, base and high are accepted as values). The best–worst range is the joint effect of every variable moving together, which the one-way analysis cannot show, and both ends are improbable because every variable would have to move the same way at once; the Monte Carlo section gives the distribution between them.

Break-even values

When a target output is entered (the profit of the alternative use of the money, a required margin, zero NPV) the tool finds for each variable the value, the others at base, at which the output equals the target, by bisection on the sweep when it crosses the target inside the range and is monotone there. Reported with it is the move from the base value as a percentage: the variable that needs the smallest move to reach the target is the one whose estimate most needs confirming before the decision. In the Eagle Airlines case the target is the $4,200 that the $52,500 down payment would earn in the money market, and the break-even hours flown (about 664) is where Clemen and Reilly's one-way graph of profit against hours crosses that line.

Global sensitivity: variance-based indices

The deterministic methods vary one variable at a time about a point. The global methods vary every variable at once over its whole distribution and apportion the variance of the output among them. With independent inputs the variance decomposes (Sobol' 1993) as V(y) = Σ V_i + Σ V_ij + … + V_12…k, and two indices summarise it for each variable: the first-order index S_i = V_i/V = V[E(y | x_i)]/V, the share of the variance that would be removed by learning x_i exactly, and the total index S_Ti = E[V(y | x_−i)]/V, the share that remains when everything but x_i is known, which counts every interaction x_i takes part in. S_Ti ≥ S_i; the difference is the interaction share of that variable; ΣS_i = 1 for an additive model and less when interactions carry variance; a variable with S_Ti near zero can be fixed anywhere in its range without changing the output distribution.

The tool estimates the indices by the Saltelli (2010) design: two independent sample matrices A and B of N rows over the k variables that carry a distribution, and for each i the matrix AB_i, which is A with its i-th column taken from B. The model is evaluated on all of them, N(k + 2) evaluations, and

S_i ≈ (1/N) Σ_j f(B)_j [f(AB_i)_j − f(A)_j] / V,     S_Ti ≈ (1/2N) Σ_j [f(A)_j − f(AB_i)_j]² / V,

with V the sample variance of f(A) and f(B) together; these are the estimators (b) and (f) of Table 2 in Saltelli et al. (2010), due to Jansen (1999), which that paper recommends. The sampling is plain Monte Carlo from the Mersenne Twister with the entered seed, so a run is repeatable, and the uncertainty of each index is shown as a 95 % bootstrap percentile interval over the rows (Archer, Saltelli and Sobol' 1997). An index below about 2/√N is within the sampling noise, and a slightly negative estimate means zero. Variables without a distribution are held at their base value and take no part. The tool's test script reproduces the analytic indices of the Ishigami function (a = 7, b = 0.1: S₁ = 0.3139, S₂ = 0.4424, S₃ = 0, S_T1 = 0.5576, S_T2 = 0.4424, S_T3 = 0.2437) and of the Sobol' g-function (a = 0, 1, 4.5, 9, 99, 99, 99, 99: S₁ = 0.716, S₂ = 0.179, S₃ = 0.0237, S₄ = 0.0072) within the sampling error, and an independent numpy run of the same estimator is filed with the sources. The Ishigami function is the standard illustration of what the one-way methods miss: x₃ has a first-order index of zero (on its own it does nothing) and a total index of 0.24, because it acts only through its interaction with x₁.

Distributions

The fifth cell of a variable line names its distribution for the global analysis, with two optional parameters: uniform a b; triangular a b with the mode at the base value; pert a b, the beta-PERT with the mode at the base value (mean (a + 4m + b)/6, λ = 4); normal mean sd; lognormal mean sd (the mean and standard deviation of the quantity itself). Blank parameters take the row's low and high, and for the normal and lognormal the mean is the base value and the standard deviation (high − low)/6, which reads the range as mean ± 3σ. Triangular and PERT values are drawn by their inverse distribution functions and from two gamma variates (Marsaglia and Tsang 2000); normal values by the polar Box–Muller method. The inputs are sampled independently: the Sobol' decomposition requires it, and a correlated analysis belongs to a different method.

Regression and rank-correlation measures

Two cheaper global measures are computed on the A sample. The standardized regression coefficient SRC_i = b_i s_xi/s_y is the coefficient of the least-squares fit y = b₀ + Σ b_i x_i scaled to standard-deviation units; for a linear additive model SRC_i² = S_i and the R² of the fit is 1, and the R² printed with it says how far the model is from that case (Helton and Davis 2003): when R² is above about 0.9 the SRC ranking is trustworthy and carries the sign of the effect, which the Sobol' indices do not; when it is well below, the model is nonlinear and the variance-based indices are the ones to use. The Spearman rank correlation ρ_i between y and x_i is the correlation of their ranks, robust to monotone nonlinearity and to outliers, and also signed.

Reading the three rankings together

The summary table puts the tornado rank, the elasticity rank and the total-index rank side by side. They answer different questions, the effect of the stated range one at a time, the local slope regardless of range, and the share of the variance under the stated distributions with everything varying, and they can disagree: a variable with a wide range and a small slope leads the tornado but not the elasticities; a variable that acts only through interactions has a small swing and a large total index. A variable near the top of all three is the one to study, measure, negotiate or control; one near the foot of all three can be fixed at base.

Worked examples on the page

The default is the Eagle Airlines case of Clemen and Reilly, Making Hard Decisions, Chapter 5: Dick Carothers considers buying a five-seat aircraft and models the annual profit from the hours flown, the charter and ticket prices, the occupancy of scheduled flights, the proportion of charter flights, the operating cost per hour, the insurance, the proportion financed, the interest rate and the purchase price. With every variable at base the profit is $9,975; with every variable at its low value −$15,927.50, at its high value $51,350; the occupancy rate and the operating cost head the tornado, and the money-market alternative yields $4,200 (the break-even target). The distributions in the variable table are illustrative additions for the global part (the book's chapter is deterministic). The example list also loads the Ishigami function and the Sobol' g-function with their analytic indices, an illustrative six-driver NPV model, and the Goferbroke EMV line of Hillier and Lieberman, whose break-even prior probability of oil (EMV of drilling = $90 of selling at p = 0.2375) the Decision Analysis tool on this site reaches by its own route.

Limits

• One output is analyzed at a time (the last formula); intermediates are shown in the scenario table only.
• The inputs of the global analysis are independent; correlated inputs need the Uncertainty Propagation tool's Gaussian copula and a different decomposition.
• The sampling is plain Monte Carlo, not a quasi-random sequence, so the indices converge as 1/√N; N is capped at 20,000 rows (N(k + 2) evaluations, about 60,000 per second).
• No second-order indices; the interaction share is reported as S_Ti − S_i per variable and 1 − ΣS_i overall.
• Two-way sensitivity graphs (the indifference line of two variables) are not drawn; the break-even table gives the one-variable crossings.