Reliability Demonstration Test: Method

Reliability Demonstration Test: Success Run, MTBF Demonstration, Sequential Test, Weibull Substantiation, Acceleration and Cost

What a demonstration test proves

A reliability requirement reads "reliability R at time t, demonstrated with confidence C". A demonstration test is a pass / fail rule fixed before the test starts: so many units, run so long, passed if no more than f failures occur. The rule demonstrates the requirement when a product that is exactly at the requirement would pass with a probability of no more than 1 − C. That probability is the consumer's risk β; the confidence is its complement. A passed test therefore says: had the product been at the requirement or worse, a pass like this one would have been unlikely. The other risk is the producer's: the probability α that a product with the reliability the design is expected to have fails the test by bad luck. The zero-failure plans fix β and leave α to fall where it may; the fixed-duration and sequential plans of MIL-HDBK-781A fix both, at two reliability levels, and pay for it in test time. The operating characteristic (OC) curve, the probability of passing against the true reliability, shows both risks at once and is drawn for every plan.

Success run (binomial)

Each of n units runs one mission and either survives or fails; nothing is assumed about the life distribution. If the true reliability is R, the number of failures is binomial and the test is passed with probability P = Σ_{i ≤ f} C(n, i) (1 − R)ⁱ R^{n − i}. With f = 0 this is Rⁿ, and Rⁿ ≤ 1 − C gives the success-run formula n ≥ ln(1 − C)/ln R: 22 units for R = 0.90 at 90 % confidence, 45 for 0.95, 230 for 0.99. With failures allowed the tool finds the smallest n for which P ≤ 1 − C by the exact binomial (38 units for one failure and 52 for two at 0.90 / 90 %). After a test of n units with f failures the one-sided Clopper-Pearson lower bound R_L is the reliability at which f or fewer failures has probability 1 − C; in terms of the beta distribution R_L is the (1 − C) quantile of Beta(n − f, f + 1), and with no failure R_L = (1 − C)^1/n. The confidence that the reliability is at least a stated R is 1 − P(f or fewer failures | R). Both are exact for pass / fail data and conservative in the sense that the true coverage is at least the stated confidence.

Exponential MTBF demonstration

When the failure rate is constant (repairable equipment in its useful life, the assumption of MIL-HDBK-781A), reliability at t is exp(−t/θ), the requirement becomes an MTBF θ = t/(−ln R), and failures in a total test time T are Poisson with mean T/θ. Only the total time on all units matters: one unit for 1,000 hours is ten units for 100 hours. A test stopped at a planned time T with r failures gives the lower bound θ_L = 2T/χ²(C; 2r + 2); a test stopped at the r-th failure gives 2T/χ²(C; 2r); the upper bound is 2T/χ²(1 − C; 2r) in both cases, χ²(p; ν) being the value below which a chi-square variable with ν degrees of freedom falls with probability p. Turned round, the test time that demonstrates θ₁ with r or fewer failures is T = θ₁ χ²(C; 2r + 2)/2; the factor χ²(C; 2r + 2)/2 is the Test Length Guide Table of the NIST/SEMATECH e-Handbook (2.30 for no failure at 90 %, 3.89 for one, 5.32 for two). With no failure the factor is −ln(1 − C), and T/t = ln(1 − C)/ln R is the success-run number again: under the exponential assumption 21.85 mission lives of failure-free running demonstrate 0.90 at 90 %, however they are divided among units.

Fixed-duration plans with both risks. MIL-HDBK-781A names the MTBF to be demonstrated the lower test MTBF θ₁ and the design value the upper test MTBF θ₀ = d θ₁, d being the discrimination ratio. A plan (T, a) tests for T and accepts with a or fewer failures; β = P(N ≤ a | T/θ₁) and α = 1 − P(N ≤ a | T/θ₀). Both nominal risks can be met only when χ²(1 − β; 2a + 2) ≤ d χ²(α; 2a + 2); the tool takes the smallest such a and the shortest T, so its β is exact and its α is at or below the nominal. The handbook's twelve standard plans (IX-D to XVII-D and the high-risk XIX-D to XXI-D, Tables XI and XII) are shown with their printed true risks and the values recomputed from the Poisson distribution, which agree to the printed 0.1 %. Those plans accept a slight excess on one risk to keep the test short: plan XIV-D (d = 2, nominal 20 % / 20 %) runs 7.8 θ₁ with true risks 19.9 % and 21.0 %, where the strict plan needs 9.08 θ₁. A small d makes long tests: at d = 1.5 and 10 % risks the handbook plan runs 45 θ₁.

Probability ratio sequential test

Wald's sequential test decides as the data arrive. For the exponential the likelihood ratio of θ₁ against θ₀ after r failures in total time t is (θ₀/θ₁)^r exp[−(1/θ₁ − 1/θ₀) t]; the test continues while it lies between B = β/(1 − α) and A = (1 − β)/α. Taking logarithms gives two parallel lines in the plane of failures against time (time in multiples of θ₁): continue while a + b t < r < c + b t, with a = ln B/ln d, c = ln A/ln d and b = (1 − 1/d)/ln d; accept on the lower line, reject on the upper. MIL-HDBK-781A (section 5.9.3) multiplies A by (d + 1)/(2d), a correction that offsets the effect of truncation on the risks, and truncates the test (5.9.4): the smallest chi-square degrees of freedom ν with χ²(α; ν)/χ²(1 − β; ν) ≥ 1/d gives r₀ = ν/2 rounded up, and T₀ = θ₀ χ²(α; 2r₀)/2. The handbook's worked case (α = β = 0.10, θ₁ = 100 h, θ₀ = 200 h) gives A = 6.75, a = −3.17, c = 2.75, b = 0.0072 per hour, r₀ = 15 and T₀ = 2,060 h, which the tool reproduces.

Truncation and the discreteness of failures move the true risks away from the nominal ones, so the tool computes them exactly. Between two consecutive boundary times the reject number is constant; the probabilities of being still on test with j failures advance by Poisson increments, the paths that reach the reject number leave as rejects, and at each accept time the paths at or below the accept number leave as accepts. The expected test time is the integral of the probability of still testing, in closed form on each interval. Applied to the accept-reject table the handbook prints for Test Plan III-D (Figure 11), the recursion returns 12.8 % and 12.8 %, the true risks the handbook states for that plan. The standard plans I-D to VIII-D are not the bare formula: their reject lines are shifted and their truncation points adjusted to balance the risks (III-D rejects at the third failure before 0.70 θ₁, where the formula gives 0.34 θ₁, and truncates at 16 failures, not 15). The tool draws the formula plan with the truncation it computes or the one entered, and states its exact risks, which is what a tailored plan needs. A sequential test accepts a good product and rejects a bad one sooner on average than the fixed-duration plan of the same risks, but its length is not known in advance; budget to the truncation.

Pass / fail units. For units tested one after another with failure probabilities p₀ = 1 − R₀ (design) and p₁ = 1 − R (required), Wald's lines are x = s n − h₁ (accept) and x = s n + h₂ (reject), with g = ln[p₁(1 − p₀)/(p₀(1 − p₁))], s = ln[(1 − p₀)/(1 − p₁)]/g, h₁ = ln[(1 − α)/β]/g and h₂ = ln[(1 − β)/α]/g. The risks and the average sample number are computed by exact recursion over the units up to the truncation, where the test accepts if the failures are at or below s n.

Weibull zero-failure test and Weibayes

Wear-out failure modes are not exponential, and a test that runs units longer than the mission says more than the pass / fail count admits. If the mode is Weibull with a shape β known from earlier plots of the same mode, R(t) = exp[−(t/η)^β] and the requirement is a characteristic life η = t/(−ln R)^1/β (AFWAL-TR-83-2079 eq. 5.2). When n units each run t_w without failure the test is passed with probability exp[−n (t_w/η)^β]; setting this to 1 − C gives t_w = η [−ln(1 − C)/n]^1/β, the characteristic life multipliers of the report's Table 5.1, and n = −ln(1 − C)/(t_w/η)^β, its Table 5.2. In mission lives, t_w/t = [ln(1 − C)/(n ln R)]^1/β: at one mission life n is the success-run number, and for β = 2 ten units run 1.48 lives replace 22 units run one life. For β above 1 fewer units run longer need less total test time; for β below 1 the reverse; at β = 1 the total is constant. With f failures allowed the tool solves the binomial for the failure probability p by the end of test at which P(pass) = 1 − C and sets t_w = η [−ln(1 − p)]^1/β. The report's section 5.7 plan, which also holds the probability of passing at a design life, is in the library and the test script.

The assumed shape carries the result. A test of a few units run for several lives is short on hardware and leans entirely on β and on the units being typical of production; run the tool over the range of shapes the history supports, and keep to at least three units unless there is reason not to. Extrapolation beyond about twice the longest proven life also assumes that no second failure mode appears.

Weibayes. With β known, t^β is exponential with mean η^β, so the chi-square bound applies to Σ t_i^β over all units, failed or not: η_L = [Σ t_i^β/(χ²(C; 2r + 2)/2)]^1/β, and with no failure the divisor is −ln(1 − C). The report's Weibayes line assumes the first failure is imminent (divisor 1), which is the zero-failure bound at 63.2 % confidence (its Appendix E). The demonstrated reliability at the mission time is exp[−(t/η_L)^β].

Acceleration

If the test condition is more severe than use and shortens life by a factor AF without changing the failure mode or the shape, one hour of test counts as AF hours of use and every test time is divided by AF. The tool takes an entered factor, the Arrhenius factor exp[(E_a/k)(1/T_use − 1/T_test)] with absolute temperatures and Boltzmann's constant k = 8.617333 × 10⁻⁵ eV/K, or the inverse power law (S_test/S_use)ⁿ. The factor is an input to the demonstration, not a product of it: the activation energy or exponent must come from tests of the same failure mode (the Accelerated Life tool fits them), and the confidence stated does not cover the uncertainty of AF.

Cost and choice of plan

The cost of a plan is taken as units × cost per unit + total test time × cost per unit of time, at the test condition. The table lists every plan computed; the search runs the zero- to three-failure Weibull plans (β = 1 when only the exponential applies) over the allowed range of units and an optional time limit per unit, and reports the cheapest. Cheapest is not the same as best: the zero-failure plan has the lowest cost and the highest chance of failing a good product (the tables give the probability of passing at the design level), and a plan with one or two failures allowed, or a sequential plan, is the usual remedy when that chance is too high.

Checks and sources

The library is checked against the NIST e-Handbook test-length table (78 cells), the twelve fixed-duration plans and the confidence-multiplier tables of MIL-HDBK-781A, its Figure 18 and 19 plan tables, its sequential example and Test Plan III-D, Tables 5.1 and 5.2 and the worked examples of AFWAL-TR-83-2079, scipy on twenty seeded plans, and seeded Monte Carlo runs of the truncated sequential tests. Printed cells that the mathematics does not reproduce are recorded in the test script: three multipliers in the handbook's Tables XIII and XIV, one row of its Figure 18, one cell of the report's Table 5.2, and two sample sizes in the report's problem 5-3. All three documents are public (U.S. Department of Defense, NIST, U.S. Air Force); the equations of Wald, Epstein and Sobel, and Clopper and Pearson are used as mathematics.