IIID. Assessing Reliability Growth

In RDT, as in the entire TAAF process, the primary focus is on accomplishing reliability improvement rather than demonstrating reliability achievement. However, because of the relatively formal control, RDT is better suited to quantification than other elements of the TAAF process. RDT can help answer the quantitative questions of where are we? (currently achieved reliability) and how rapidly are we progressing? (growth rate). In this section, we discuss the most important models, interpretation of data, and the kinds of decisions you may need to make that growth assessment will support.

The only direct, model-independent measure of current reliability is the elapsed time between the two most recent failures. The uncertainty associated with a sample of one is so large as to make that measure virtually useless. Therefore, it is necessary to take the accumulated data in RDT into account, which in turn requires choosing a model for the growth process.

MIL-HDBK-189 discusses a number of growth models, with emphasis on the AMSAA model. Section 4.3 of MIL-HDBK-781 indicates the procedures to be followed in employing the AMSAA and Duane methods. The AMSAA model is essentially the stochastic extension of Duane's method. Both model reliability growth as a power function of test time. The Duane method, with its heavy reliance on graphic techniques, is well suited for quick (and "dirty") analyses and for detection of discontinuities in the growth process. A Duane plot requires only logarithmic graph paper and the calculation of ratios of cumulative test time to cumulative failures. The AMSAA model is essential for determination of confidence bounds and for objectivity in parameter estimation. It should always be used in connection with contractual quantification requirements. Use of the AMSAA model requires a few minutes with a calculator or a few seconds on a personal computer.

Both models imply continuity of growth, which requires
that testing be suspended after each failure until the corresponding fix has
been implemented in all test items. In the real world, discontinuous growth is
the norm. In fact, subparagraph 202.2.2.3 of MIL-HDBK-189 explicitly permits
the procuring activity to authorize replacement of failed items so that the
test can continue while failures are being investigated. Figure 5 (adopted
from MIL-HDBK-189, Figure 5.29) illustrates the effects of such test
continuation. Figure 6 (from MIL-HDBK-189, Figure 5.30) also shows the impact
of noninstantaneous fixes on the relationship between test time and calendar
time. Although such discontinuities interrupt the smooth theoretical progress
of reliability growth versus test time, it is common (and reasonable) practice
to make estimates as if a smooth power function were the appropriate model
unless there is clear evidence that distinct test phases need to be analyzed
separately. Plotting methods are well covered in MIL-HDBK-189.

Reliability growth produces inherently very "noisy"
test data--even under ideal conditions. This fact is reflected in the wide
confidence intervals tabulated in Appendix
C
of MIL-HDBK-189. Appendix C of this technical brief presents additional information not covered by MIL-HDBK-189. After 5 failures, 10 percent of the MTBF estimates can exceed the true value by factors greater than 2.6, and 10 percent at less than one half of the true value. Even at the 30th failure--implying a fairly lengthy test--the factors can be approximately 1.4 and 0.7, respectively.

Growth rates are also difficult to measure accurately.
They can be in error by a factor of 3 or more in either direction for
reasonable test times. It is possible for the growth to appear as negative
(that is, worsening) when the true trend is positive and vice versa. The
possibility for error varies with growth rate (relatively much smaller for
higher growth rates) (See
Appendix C
). Thus, estimated reliability and estimated growth aid your judgment--they do not substitute for it. Attempts at quantification of RDT results for contractual purposes should be avoided unless a test program of substantial length is implemented.

The types of decisions you may need to make and for which growth assessment is an aid are the following:

Early termination of RDT on the basis of compelling evidence (on the growth curve) that reliability objectives have been reached early.

Early termination of RDT when quantitative and qualitative results (that is, types and patterns of failure) indicate a need for major redesign (followed by a TAAF restart) rather than a series of fixes.

Extension of RDT when there is both substantial doubt that the objectives have been reached and confidence that growth will continue--for example, when there is evidence of growth, but at a lower rate than planned.

Extension of RDT when additional time is needed to verify the effectiveness of the most recent fixes.