Cost Risk Analysis: Method

Distributions, Correlation, Monte Carlo S-Curve and Contingency

Why a point estimate is not enough

A cost estimate built from the most likely value of every work breakdown structure (WBS) element is a single number that will almost never be the final cost. Each element is uncertain, and cost distributions are skewed to the right: there are more ways for an element to cost more than planned than less, so the mean of each element lies above its most likely value and the sum of the most likely values sits well below the median of the total. The GAO Cost Estimating and Assessment Guide (GAO-20-195G, Chapter 12) and the NASA Cost Estimating Handbook (Appendix G) therefore ask that every estimate carry a risk and uncertainty analysis: a probability distribution of the total cost, the cumulative probability of the point estimate on it, and the contingency needed to reach the confidence level management chooses. This tool performs that analysis for a WBS pasted as text.

Element distributions

Each element is entered either as three points (low a, most likely m, high b) with a distribution shape, or as a mean and standard deviation (the layout of NASA CEH Table G-4) for a normal or lognormal element. The shapes and their moments are:

• Triangular (a, m, b): mean (a + m + b)/3, variance (a² + m² + b² − am − ab − mb)/18, mode m. The usual shape for expert three-point estimates (GAO Chapter 12, NASA Figure G-4).
• PERT: the four-parameter beta on [a, b] with shape parameters 1 + 4(m − a)/(b − a) and 1 + 4(b − m)/(b − a), so that the mean is (a + 4m + b)/6 (Malcolm, Roseboom, Clark and Fazar 1959). Its exact variance is (μ − a)(b − μ)/7; the classical (b − a)/6 is an approximation to its standard deviation and is printed beside the exact value. The PERT puts less weight in the tails than the triangular with the same three points.
• Uniform (a, b): mean (a + b)/2, variance (b − a)²/12; for an element known only to lie in a range.
• Normal: from a mean and standard deviation, or from a low and high read as percentiles: μ = (a + b)/2, σ = (b − a)/(2z) with z the standard normal quantile of the upper percentile (1.2816 for 10/90, 1.6449 for 5/95, 1.0364 for 15/85).
• Lognormal: from a mean and standard deviation (σ_ln² = ln(1 + σ²/μ²), μ_ln = ln μ − σ_ln²/2) or from percentiles (μ_ln = (ln a + ln b)/2, σ_ln = (ln b − ln a)/(2z)); mean e^{μ_ln + σ_ln²/2}, mode e^{μ_ln − σ_ln²}. NASA's Appendix G prefers it over the normal because it cannot go negative and is right-skewed.
• Fixed: a = b or a zero standard deviation; a firm quote or a contracted price.

The page tabulates the mode, mean, standard deviation and the 10th and 90th percentiles of every element so that the shapes can be compared on one footing.

Low and high as bounds or as percentiles

Experts asked for the lowest and highest possible cost rarely give the true extremes; GAO cites studies showing that subject-matter experts capture about 70 % of the real range, and describes the practice of reading their minimum and maximum as the 15th and 85th percentiles and widening the distribution accordingly. The "Low and High Are" selector does this: with a percentile choice the triangular, PERT and uniform bounds are solved outward (two equations F(a_entered) = p, F(b_entered) = 1 − p in the unknown bounds, the mode kept) so that the entered values sit at the chosen percentiles; the normal and lognormal always read the low and high as percentiles. The widened bounds are printed in the element's note.

Analytic roll-up

The mean of the total is the sum of the element means whatever the correlation. Its variance is σ² = Σ_iΣ_j ρ_ij σ_i σ_j: the root-sum-square √Σσ_i² when the elements are independent, the plain sum Σσ_i when every pair is perfectly correlated, and in between otherwise. With a common coefficient ρ between every pair, σ² = (1 − ρ)Σσ_i² + ρ(Σσ_i)². By the central limit theorem the sum of many elements approaches a normal distribution (NASA Figure G-3), so μ + zσ gives a first approximation to each percentile; the simulation shows how far the skew of the inputs, the register events and the correlation move the real percentiles away from it. The NASA Appendix G example on the page (eleven lognormal elements, correlation 0.6 everywhere) has a mean of $236.3M and a standard deviation of $26.9M, the handbook's "$236 million" with one standard deviation at "$209 million" and "$263 million".

Correlation

Cost elements move together: the same labour rates, the same schedule, the same design maturity. Positive correlation does not change the mean of the total but widens its distribution, so it matters most at the high percentiles where the contingency is read (GAO Figure 17: adding 0.9 correlation moved the 50th percentile by $1M and the 80th by $24M). GAO's rule is that correlation should never be ignored; the Joint Agency Cost Schedule Risk and Uncertainty Handbook recommends 0.3 as a default when nothing better is known, NASA's example uses 0.6. The tool applies one coefficient within correlation groups (a label after the numbers of a row), or one coefficient between every pair, or a pasted matrix (lower triangle or full). A matrix that is not positive definite (impossible as a set of correlations) is replaced by the nearest correlation matrix by eigenvalue clipping (Higham 2002) and the page says so. Elements with a fixed cost are uncorrelated by nature.

Monte Carlo simulation

In each iteration every element is drawn from its distribution, each register risk occurs with its probability and then draws its impact, the schedule delay is drawn and multiplied by the burn rate, and everything is summed. The sorted totals of all iterations form the empirical cumulative distribution, the S-curve, from which the percentiles, the cumulative probability of the point estimate and the contingency are read. Two methods induce the correlation:

• Iman–Conover (1982) rank correlation, the method of @RISK and Crystal Ball: every element is sampled independently, then the samples are rearranged to follow the rank order of a matrix of correlated van der Waerden scores whose sample correlation is made exactly the target (Cholesky of the target and of the score correlation). The marginal distributions are kept exactly and the rank correlation of each pair comes out at its target to within sampling error; the page prints the achieved rank correlation of the most strongly correlated pair as a check.
• Gaussian copula: correlated standard normal scores are generated by the Cholesky factor of the target matrix and each is mapped through its element's inverse distribution function. The Pearson correlation of the outputs is slightly below the target for skewed marginals.

The simulation is seeded, so the same inputs and seed reproduce the same result; 10,000 iterations put the P80 within about 0.3 % of its converged value for the examples on the page, and 50,000 are allowed.

Contingency and confidence levels

GAO defines the contingency as the difference between the point estimate and the cost at the confidence level management selects, and asks the estimator to report the cumulative probability of the point estimate itself: a point estimate at the 15th percentile, as in GAO's UAV example, has only a 15 % chance of not being exceeded. No single level is a best practice; budgeting to the mean is common, 70 or 80 % is used for high-risk programs (NASA policy sets a joint cost and schedule confidence level of 70 % for its projects), and a portfolio funded at a high level everywhere may be unaffordable. The page gives the cost and the contingency at every level entered, against the point estimate, the sum of the element means and the sum of the most likely values; the P80 − P50 difference, a common management-reserve convention above a median budget; and GAO's two allocations of the contingency to the elements, proportional to each element's standard deviation or to its "need" (its own cost at the level minus its point value), both scaled to the simulated total so that allocating element percentiles one by one does not overstate the reserve.

Risk register and schedule-driven cost

Discrete risks (a supplier slips, a test fails) are not well represented by widening an element's range. The register takes each as a Bernoulli event with probability p and an impact distribution: its expected monetary value is p × mean impact (PMBOK Guide), its unconditional variance p(σ_I² + μ_I²) − (pμ_I)². Such an event adds little to the mean but a step to the upper tail, which the S-curve shows. A schedule-driven cost models the standing cost of the project during a delay: days of delay (low, most likely, high) times a burn rate per day, independent of the elements.

Risk drivers and the tornado

Three measures rank the items by their influence on the total. The variance share cov(X_i, T)/var(T), analytic from the input σ and ρ and simulated from the sample, sums to 100 % and includes each item's covariance with the others, so with correlation an item's share includes the part of the other elements that moves with it. The Spearman rank correlation of each item with the total is the sensitivity measure of the commercial tools. The tornado swings each item alone from its 10th to its 90th percentile with every other item at its point value and plots the resulting totals as horizontal bars, largest swing on top. The items at the top are where better data, a firm quote or a design decision reduces the contingency most.

Reading the results

• The cumulative probability of the point estimate: below 50 % the estimate is optimistic; the lower it is, the larger the contingency any realistic budget needs.
• The gap between the sum of the most likely values and the sum of the means is the part of the contingency that is simply the skew of the inputs, before any correlation or discrete risk.
• Mean versus median: a mean above the median marks a right-skewed total; the skewness is printed.
• The analytic normal percentiles beside the simulated ones show how much the shape of the inputs matters; they agree closely when the elements are many, symmetric and independent.
• Correlated standard deviation against RSS: the percentage increase is the price of the dependence between elements; it is what a rank correlation of 0.3 or 0.6 is worth at the P80.

Worked example on the page

The default case is the NASA Cost Estimating Handbook Version 4.0, Appendix G, section G.5: a flight-system WBS of eleven Level 2 elements with the means and standard deviations of Table G-4 (lognormal distributions), every pair correlated at 0.6 (Table G-5). The handbook prints a mean of $236 million with one standard deviation below and above at $209 million and $263 million, and reads from its S-curve a 70 % chance that the cost is below $248 million; the analytic roll-up gives 236.3 and 26.9, and the simulation puts the 70th percentile at 248. (The handbook's text also quotes a 40 % chance below $221 million, a reading of its figure that neither the normal approximation nor the simulation reproduces, both giving about 227 to 229.) The second example of the selector is an illustrative twelve-element WBS with three correlation groups, four register risks and a schedule-driven cost; its values are not from a publication. The test script checks the library against the handbook's numbers, GAO's printed contingency arithmetic, the closed forms of every distribution and an independent numpy run of two million draws.

Limits of the tool

• The register risks and the schedule term are independent of the elements and of each other; a risk that drives several elements is better modelled as a correlation group or by widening those elements (GAO's risk-driver method is not implemented).
• Correlation is applied as a Pearson coefficient (analytic) and as a rank or copula correlation (simulation); for skewed elements the two differ slightly.
• Percentile-defined bounds assume the most likely value lies between the entered low and high.
• No time phasing, inflation or discounting: every cost is in one currency unit of one year. No schedule network: the schedule term is a single delay distribution.
• Contingency allocation follows GAO's two proportional rules; the allocations are shares of one simulated total and do not represent a commitment to fund the elements.