This is the accessible text file for GAO report number GAO-12-162
entitled 'International Space Station: Approaches for Ensuring
Utilization through 2020 Are Reasonable but Should Be Revisited as
NASA Gains More Knowledge of On-Orbit Performance' which was released
on December 15, 2011.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as
part of a longer term project to improve GAO products' accessibility.
Every attempt has been made to maintain the structural and data
integrity of the original printed product. Accessibility features,
such as text descriptions of tables, consecutively numbered footnotes
placed at the end of the file, and the text of agency comment letters,
are provided but may not exactly duplicate the presentation or format
of the printed version. The portable document format (PDF) file is an
exact electronic replica of the printed version. We welcome your
feedback. Please E-mail your comments regarding the contents or
accessibility features of this document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
[Note: This report was revised on March 19, 2012, to correct the legend
for Figure 4 on page 23]
United States Government Accountability Office:
GAO:
Report to Congressional Committees:
December 2011:
International Space Station:
Approaches for Ensuring Utilization through 2020 Are Reasonable but
Should Be Revisited as NASA Gains More Knowledge of On-Orbit
Performance:
GAO-12-162:
GAO Highlights:
Highlights of GAO-12-162, a report to congressional committees.
Why GAO Did This Study:
In 2010 the National Aeronautics and Space Administration (NASA) was
authorized to extend the life of the International Space Station (ISS)
from 2015 through at least September 30, 2020. Gauging the feasibility
of doing so is quite complex. Among the factors to be assessed are the
reliability of key components, NASA’s ability to deliver spares to the
ISS, the projected life of structures that cannot be replaced, and the
health of systems that affect safety. While some empirical data exist,
the ISS is a unique facility in space and assessing its extended life
requires the use of sophisticated analytical techniques and judgments.
GAO provided a preliminary report on NASA’s use of such techniques in
April 2011. For this review, GAO assessed the extent to which NASA has
ensured essential spare parts are available and ISS structures and
hardware are sound for continued ISS utilization through 2020. GAO
interviewed NASA officials and outside experts; assessed the
methodology underlying NASA’s findings; conducted a limited test of
data supporting NASA’s assessments; and analyzed documentation such as
ongoing assessments, schedules and other relevant efforts.
What GAO Found:
NASA’s approach to determining, obtaining, and delivering necessary
spare parts to the ISS is reasonable to ensure continued utilization
of the station through 2020. The statistical process and methodology
being used to determine the expected lifetimes of replacement units is
a sound and commonly accepted approach within the risk assessment
community that considers both manufacturers’ predictions and the systems
’ actual performance. To date NASA has given equal weight to
manufacturers’ predictions and actual performance, and currently has
no plans to reassess this decision. However, as time goes on, the
resulting estimates could prove to be overly conservative, given that
NASA has found failure rates for replacement units to be lower than
manufacturers’ predictions. Therefore, continuing to weigh the
manufacturers’ predictions equally with actual performance could lead
NASA to purchase an excess of spares. NASA also has a reasonable
process for establishing performance goals for various functions
necessary for utilization and determining whether available spares are
sufficient to meet goals through 2020, but the rationale supporting
these decisions has not been systematically documented.
Figure: International Space Station:
[Refer to PDF for image: photograph]
Source: NASA.
[End of figure]
NASA is also using reasonable analytical tools to assess structural
health and determine whether ISS hardware can operate safely through
2020. On the basis of prior analysis of structural life usage through
2015 and the robust design of the ISS structures, NASA currently
anticipates that—with some mitigation—the ISS will remain structurally
sound for continued operations through 2020. NASA also is using
reasonable methodologies, governed by agency directives and informed
by NASA program experts, to assess safe operations of the ISS as a
whole as well as identify replacement units and other hardware,
failure of which could result in an increased risk of loss of station
or loss of crew through 2020. NASA plans to develop, through 2015,
methods to mitigate issues identified and expects to begin
implementing corrective actions as plans are put in place.
What GAO Recommends:
GAO recommends that NASA reassess the relative weight given to
original reliability estimates of spares’ life expectancies as
performance data accumulates and document the rationale behind
specific performance goal decisions. NASA concurred with our
recommendations.
View [hyperlink, http://www.gao.gov/products/GAO-12-162]. For more
information, contact Cristina Chaplain at (202) 512-4841 or
chaplainc@gao.gov.
[End of section]
Contents:
Letter:
Background:
NASA's Sparing Approach to Ensure Continued ISS Operations and
Utilization through 2020 Employs Appropriate and Reasonable
Methodologies:
NASA Is Using Reasonable Methods to Assess Structures and Hardware:
Conclusion:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Scope and Methodology:
Appendix II: Comments from the National Aeronautics and Space
Administration:
Appendix III: GAO Contact and Staff Acknowledgments:
Tables:
Table 1: Successive Annual Updates of MTBF (in Hours) for the ISS Pump
Module Assembly:
Table 2: Examples of ISS Functions Included in a Functional
Availability Risk Assessment:
Table 3: NASA's Planned Vehicle Launches for 2012 to 2020 to Resupply
the ISS:
Table 4: Cost Estimates for ISS Hardware Spares to be Used through
2015:
Figures:
Figure 1: ISS Primary Structures, Thermal Control Functional System,
and ORU:
Figure 2: S-Band Functional Architecture:
Figure 3: Hypothetical Example of the Relationship between Original
and Updated Probability Distributions:
Figure 4: Cargo Capability of U.S. Commercial and International
Partner Vehicles Versus NASA's ISS Sparing Needs from 2012 to 2020:
Figure 5: P6 Truss in Interim and Permanent Locations:
Abbreviations:
ATV: Automated Transfer Vehicle:
BSP: baseband signal processor:
COTS: Commercial Orbital Transportation Services:
DOD: Department of Defense:
ECLSS: Environmental Control and Life Support System:
FGB: Functional Cargo Block:
HTV: H-II Transfer Vehicle:
ISS: International Space Station:
MTBF: mean time between failure:
MADS: Modeling and Analysis Data Set:
NASA: National Aeronautics and Space Administration:
ORU: orbital replacement unit:
Orbital: Orbital Sciences Corporation:
PRA: Probabilistic Risk Assessment:
SASA: S-Band antenna and support assembly:
SpaceX: Space Exploration Technologies:
TDRSS: Tracking and Data Relay Satellite System:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
December 15, 2011:
The Honorable John D. Rockefeller IV:
Chairman:
The Honorable Kay Bailey Hutchinson:
Ranking Member:
Committee on Commerce, Science, and Transportation:
United States Senate:
The Honorable Ralph Hall:
Chairman:
The Honorable Eddie Bernice Johnson:
Ranking Member:
Committee on Science, Space and Technology:
House of Representatives:
From 1994 through 2010, the National Aeronautics and Space
Administration (NASA) estimates it directly invested over $48 billion
in development and construction of the on-orbit scientific laboratory,
the International Space Station (ISS). To take advantage of that
investment, the NASA Authorization Act of 2010 extended the mission of
the ISS by a minimum of 5 years, from 2015 through at least September
30, 2020. Traditionally, the ISS received the majority of its supplies
and spare parts via the space shuttle. The space shuttle conducted its
final flight in July 2011 and as a result NASA now plans to rely on a
combination of foreign and commercially developed domestic launch
vehicles to supply hardware spares and supplies to the ISS through
2020.
In consideration of the shuttle's retirement, section 503 of the NASA
Authorization Act of 2010, directed NASA to assess its plans for
ensuring essential spares were available to support the ISS through
its planned operational life, and to report to Congress on that
assessment.[Footnote 1] The same law also directed that GAO provide an
evaluation of the accuracy and level of confidence in the findings
contained in NASA's assessment of the essential modules, operational
systems and components, structural elements, and permanent scientific
equipment required to ensure complete, effective and safe functioning
and full scientific utilization of the ISS through 2020.
Gauging the feasibility of extending the life of the ISS is no simple
exercise. Among the many factors to be assessed are the reliability of
key components, NASA's ability to deliver spares to the ISS, the
projected life of structures that cannot be replaced, and in-depth
analysis of those components and systems that affect safety. While
some empirical data exists, because the ISS is a unique facility in
space, assessing its extended life necessarily requires the use of
sophisticated analytical techniques and judgments.
NASA reported to Congress on January 11, 2011, that the ISS could be
effectively maintained through 2020 within the normal range of human
spaceflight risk with a combination of existing and planned re-supply
vehicles. Subsequently, we issued our initial evaluation on April 11,
2011.[Footnote 2] We found that NASA was using analytical techniques,
physical tests, and inspections to assess ISS structures and hardware
as well as to determine spare parts needed to support safe functioning
and full utilization of the ISS through 2020. Additionally, we noted
that NASA's assessments appeared to be supported by sufficient,
accurate, and relevant underlying data. We found, however, that NASA's
estimates of ISS sparing needs were sensitive to assumptions about
hardware reliability.
As we noted in our report, these assessments were ongoing, so not all
results were available. At your request, we continued our evaluation
of NASA's plans to ensure the ISS is supported through 2020.
Specifically, we assessed (1) the extent to which NASA has ensured
spare components essential for ISS operations will be available
through 2020, and (2) the extent to which NASA has assurance that the
ISS structure and hardware will be sound through 2020 and potentially
beyond. To conduct this work, we examined NASA's processes for
quantifying ISS maintenance needs. We interviewed knowledgeable
officials and experts within and outside of NASA to evaluate and
obtain an understanding of the methodologies and analytic techniques
NASA and other organizations use in addressing supportability. We
conducted a limited test of the reliability of key data supporting
NASA's assessments. We also examined the schedules for and manifests
of planned re-supply missions and evaluated the status of ongoing
efforts to estimate the costs of required spares. We obtained and
analyzed available results of ongoing assessments of structural health
and hardware systems. See appendix I for our detailed scope and
methodology.
We conducted this performance audit from February 2011 to November
2011 in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit
to obtain sufficient, appropriate evidence to provide a reasonable
basis for our findings and conclusions based on our audit objectives.
We believe that the evidence obtained provides a reasonable basis for
our findings based on our audit objectives.
Background:
The ISS is composed of about 1 million pounds of hardware, brought to
orbit over the course of a decade. It is the largest orbiting man-made
object and was constructed to support three activities: scientific
research, technology development, and development of industrial
applications. The facilities aboard the ISS allow for ongoing research
in microgravity, studies of other aspects of the space environment,
tests of new technology, and long-term space operations. The
facilities also enable a permanent crew of up to six astronauts to
maintain their physical health standards while conducting many
different types of research, including experiments in biotechnology,
combustion science, fluid physics, and materials science, on behalf of
ground-based researchers. Furthermore, the ISS has the capability to
support research on materials and other technologies to see how they
react in the space environment. In addition to these research
activities, the ISS crew conducts preventive and corrective
maintenance to the on-board systems.
The ISS includes (1) primary structures, i.e., the external trusses
which serve as the backbone of the station and the pressurized modules
that are occupied by the ISS crew, and (2) functional systems
comprised of orbital replacement units (ORU), i.e., systems that
provide basic functionality such as life support and electrical power
which are comprised of modular components that are easily replaced by
astronauts on orbit.
NASA originally planned for an ISS lifespan of 15 years--5 years to
assemble the individual components of the ISS on-orbit and 10 years
for operations. To ensure structural soundness for 15 years, NASA
required ISS developers to analytically demonstrate that ISS primary
structures would remain structurally sound for four service lives;
that is, at least 60 years. This demonstration involved showing
analytically that the largest undetected crack that could exist in the
structure would not grow and cause failure of the structure in four
service lifetimes[Footnote 3] when subjected to the mechanical,
thermal, and pressure loads[Footnote 4] of the on-orbit environment.
Figure 1: ISS Primary Structures, Thermal Control Functional System,
and ORU:
[Refer to PDF for image: photograph and inset]
Depicted on the inset:
Primary structures;
Thermal control functional system;
Thermal control system ORU.
Source: NASA.
[End of figure]
Although the ISS originally was planned to operate until 2015, the
NASA Authorization Act of 2008 directed the agency to do nothing to
preclude the use of the ISS beyond 2015 and the NASA Authorization Act
of 2010 directed the agency to extend operations through at least
September 30, 2020.[Footnote 5] NASA had originally planned for the
space shuttle to serve as the means of transporting hardware and
supplies to the ISS through the end of the ISS's life. However, in
2004, President George W. Bush announced his Vision for Space
Exploration that included direction for NASA to pursue commercial
opportunities for providing transportation and other services to
support the ISS after 2010.[Footnote 6] As a result of that direction,
in July 2011, the space shuttle flew its last mission to the ISS.
NASA's current strategy for supporting the ISS relies on a mixed fleet
of vehicles, including those developed by ISS international partners
and commercial domestic launch providers, to service the ISS through
2020. International partner vehicles include the European Space
Agency's Automated Transfer Vehicle (ATV), and the Japan Aerospace
Exploration Agency's H-II Transfer Vehicle (HTV). NASA has agreements
in place with the respective European and Japanese consortiums for the
ATV and HTV to provide supplies and spares to the ISS. The domestic
commercial launch vehicles, fostered under a NASA-initiated project
known as Commercial Orbital Transportation Services (COTS), are being
developed by private industry corporations Space Exploration
Technologies (SpaceX) and Orbital Sciences Corporation (Orbital). In
2006 and 2008, NASA awarded agreements to SpaceX and Orbital for
development and demonstration of cargo transport capabilities, which
SpaceX and Orbital are providing through their respective Falcon 9 and
Taurus II launch vehicles. In late 2008, NASA awarded both companies
commercial resupply services contracts to provide supplies and spares
to the ISS through 2015. NASA plans to use the SpaceX and Orbital
Sciences launch vehicles to supply the ISS beginning in 2012 and
continuing through 2020.
Initial ISS Supportability Strategy:
NASA's strategy for supporting the ISS at the beginning of the program
was based on several key factors, including:
* a specified level of ORU design reliability;
* an ISS design with electrical and mechanical functions modularized
into ORUs for simple on-orbit removal and replacement;
* a consistent, reliable transportation service to and from the ISS in
the form of the space shuttle; and:
* an approach that used the space shuttle as the vehicle for bringing
failed ORUs to earth and flying repaired ORUs back to the ISS.
The ISS was designed and certified based on the idea that its 15-year
design life would be achieved through returning failed or degraded
ORUs to earth for repair. NASA established ISS initial estimates of
spare ORUs needed using the equipment manufacturers' original
reliability predictions, and a transportation plan that assumed space
shuttle operations would continue until the planned end of ISS life in
2015. This strategy assumed not only that the space shuttle would
provide a reliable means of delivering spares to the ISS, but also a
means of returning failed or expended hardware to earth. NASA retained
many of the original equipment manufacturers of ISS hardware to
provide sustaining engineering and repair or refurbishment
capabilities for the planned life of the ISS.
Historically, NASA's methods for determining the required number of
ISS spares used a combination of the original equipment manufacturers'
projections about the length of time an ORU would operate before
randomly failing and the professional judgment of NASA engineers. NASA
and the original equipment manufacturers express an ORU's expected
frequency of random failure as the mean time between failure (MTBF),
which predicts, in hours, the average amount of time an ORU will
operate before a random failure occurs. NASA considers the MTBF as the
primary predictive measure of an ORU's reliability and uses this
information to predict when ORUs will require maintenance to prevent
or correct failures. To calculate an MTBF, the ORU's original
equipment manufacturer would typically use reliability models that
were populated with the failure rates of the individual parts obtained
from reliability handbooks, such as Military Handbook 217F and other
authoritative sources.[Footnote 7] These models also incorporate data
on part quality and quantity as well as assumptions about the ORU's
operating temperature, operating environment, and the amount of time
it spends in an active state. According to agency officials, the MTBF
calculated by using this process would typically be provided to NASA
in a reliability report as part of the technical data package
accompanying the delivery of the ORU. Officials indicated for example,
as part of the technical data package accompanying the delivery of the
ISS ammonia/water heat exchanger, which helps regulate heat on the
ISS, Boeing prepared a reliability verification analysis report that
estimated the MTBF for this ORU is 362,401 hours.
Revised ISS Supportability Strategy:
NASA has revised its ISS supportability strategy for ORUs from the
original "repair and return" strategy to one of "build and dispose."
Three factors played major parts in this revision:
* the decision to decommission the space shuttle;
* NASA's observation that reliability of on-orbit ORUs has been much
higher, on average, than the original equipment manufacturers' initial
predictions of MTBF; and:
* NASA's recognition that procurement of enough spares to ensure full
functionality through the end of ISS life, based on manufacturers'
predictions, would be unaffordable and was not necessary to ensure
safe, useful ISS operations.
Lacking a means upon retirement of the space shuttle to return failed
ORUs to earth, repair them, and take them back to the ISS, NASA moved
to a strategy of replacing failed ORUs with new ones. NASA pre-
positioned spares on-board the ISS to ensure that the station could
operate in the intervening period between shuttle retirement and
expected development of new launch vehicles. Because NASA no longer
has the sparing flexibility provided by the space shuttle, and
astronauts can perform only minor repairs while on-orbit, NASA's
current supportability strategy relies more heavily on modeling and
computational analysis to determine the quantity of spare ORUs needed
to support the station.
NASA also revised its ISS supportability strategy because ORU on-orbit
failure data indicated that the manufacturers' MTBFs were, on average,
too conservative. According to NASA representatives, this conservatism
resulted from original equipment manufacturers' tendency to err on the
side of caution, that is, the manufacturers were primarily interested
in demonstrating through analysis that ORUs they developed would meet
minimum MTBF criteria established by NASA. Moreover, the component-
level reliability data contained in Military Handbook 217F that served
as the basis for original equipment manufacturers' analysis has also
proved to be overly conservative in many cases.
Given the actual on-orbit failure data, NASA recognized that a sparing
strategy built around manufacturers' original MTBF estimates needed to
be modified or it would result in the acquisition of many more spares
than were truly needed to support the program. Consequently, the ISS
program set up a team of experts referred to as a "tiger team" to
evaluate other methods of determining more accurate MTBF estimates for
ORUs. In 2005, this team settled on a new NASA process that used a
Bayesian estimation process to update the ORUs' original MTBFs. The
Bayesian estimation process combines the manufacturer's original MTBF
with actual on-orbit data--including the accumulated hours of
operation and random failure history of an ORU population--and results
in an updated MTBF reflecting NASA's experience with ORU
performance.[Footnote 8] In January 2006, the ISS Reliability and
Maintainability Panel approved the use of the Bayesian estimation
process to update MTBF estimates.[Footnote 9] Since that time, NASA
has implemented an annual Bayesian MTBF update process that continues
to refine reliability and failure projections for ISS ORUs.
While an improved method for updating the MTBF of each ORU was needed
to inform NASA's sparing decisions, it was not sufficient by itself to
predict the required quantity of spares needed, since the reliability
of individual ORU is only one piece of the puzzle. Groups of ORUs are
arranged into an ORU architecture to support a specific function. Many
functional architectures include redundancy within their design so
that the failure of a single ORU does not disable the function. For
example, the S-band communications and tracking function--which
supports all command and control from the ground--consists of six ORUs
arranged into two independent strings.[Footnote 10] Each string
consists of three ORUs: (1) the S-Band antenna and support assembly
(SASA); (2) the Tracking and Data Relay Satellite System (TDRSS)
Transponder; and (3) the baseband signal processor (BSP). Each string
has the capacity of being designated prime or backup using a software
switch which automatically turns on the backup if the prime fails.
Figure 2 illustrates the ORU functional architecture for the S-band
communications and tracking function.
Figure 2: S-Band Functional Architecture:
[Refer to PDF for image: illustration]
String A (Prime):
SASA;
TDRSS Transponder;
BSP.
String B (Backup):
SASA;
TDRSS Transponder;
BSP.
Switch:
controls activity between the strings.
Source: GAO analysis of NASA data.
[End of figure]
A failure of a single ORU will disable its string, but will not
disable the function since the function can be switched over to the
second string.
To enable the ISS program to predict how many spare ORUs it required,
NASA needed not only the best available estimate of each ORU's MTBF,
but also a means of quantifying what level of redundancy was required
for each function, and with what level of confidence each function
must be maintained. When it considered the challenges of supporting
ISS without the space shuttle's ability to guarantee the repair and
refurbishment of failed ORUs, the ISS program recognized that the ISS
would need to "gracefully degrade" without compromising safety as it
approached its end of life. Graceful degradation was defined by the
ISS program for 27 functions through the establishment of a
functionality target at the end of the ISS's life, and a confidence
goal for meeting that functionality target. The ISS program also
recognized that it would need an appropriate analytic method for
predicting whether it would meet its functionality targets and
confidence goals. The functionality target and confidence goal for
each function drives the quantity of spare ORUs required through the
end of the ISS's life. For example, the ISS program established a
functionality target for its S-band communications and track function
of having at least one of the two ORU strings operating at end of
life, and a 98 percent confidence goal for meeting this functionality
target. The ISS program has conducted an annual functional
availability risk assessment of 27 functions every year since 2006 to
assess whether or not the ISS is meeting its functionality targets and
confidence goals through 2015. In 2010 NASA conducted the first
assessment addressing ISS functionality through 2020. This assessment
employs a Monte Carlo model to calculate, for each ISS function, the
probability of meeting its functionality targets as a function of time
through end-of-life under different sparing quantity
assumptions.[Footnote 11] The assessment uses the results from the
Bayesian MTBF process as an input.
NASA's Sparing Approach to Ensure Continued ISS Operations and
Utilization through 2020 Employs Appropriate and Reasonable
Methodologies:
NASA has chosen appropriate and reasonable methodologies to predict
the quantity of spares it requires to maintain ISS operations and full
scientific utilization through 2020. NASA uses a Bayesian estimation
process to improve estimates of ORU MTBF and functional availability
risk assessments to determine with what confidence ISS functionality
targets can be maintained. The Bayesian estimation process is an
appropriate and reasonable technique for updating the original
predicted MTBF of ISS ORUs. Similar techniques have been used on other
NASA programs and are used throughout government and industry to
update reliability estimates for nuclear power plant components,
pressure vessels at oil and gas facilities, and medical devices.
Moreover, the ISS program's specific implementation of this process is
reasonable, given its current level of knowledge of the on-orbit
performance of ORUs. However, the results of the Bayesian estimation
process are sensitive to key assumptions, particularly with regard to
the weight given to the manufacturer's original estimated MTBF
relative to the actual on-orbit performance and failure data. The ISS
program's functional availability risk assessment process is also an
appropriate and reasonable technique for predicting whether available
spares are sufficient to allow the ISS to meet its functionality
targets and confidence goals, set by the program manager with input
from experts. Moreover, the ISS program conducts sensitivity analysis
to determine how the number of required spares would change if the
functionality targets and confidence goals were relaxed. The ISS
program manager's rationale for selecting these targets and goals,
however, has not been systematically documented.
Bayesian Estimation Process Is an Appropriate and Reasonable Technique
for Updating Estimates:
The ISS program's Bayesian estimation process employs a common
statistical technique for updating the original reliability estimates
of ISS ORUs. The process combines the probability distribution of the
original equipment manufacturer's estimated MTBF with actual on-orbit
data, which includes the accumulated hours of operation and random-
failure history of the ORU population.[Footnote 12] The result is a
probability distribution of a new "operational MTBF" reflecting NASA's
experience with the ORU. The basic idea underlying Bayesian estimation
is that an original estimate for a statistical parameter--the MTBF,
for example--can be improved upon by combining it with actual
experience. As more experience is gained and applied in the updating
process, the accuracy of the new distribution that defines the MTBF
will improve. If the original distribution has a relatively large
variance--for example, substantial uncertainty in the value of the
MTBF--then the new evidence (accumulated hours of operation and the
number of on-orbit random failures) will tend to have a larger effect
on the new distribution. On the other hand, if the original
distribution has a relatively small variance--that is, a high degree
of certainty that the value of the MTBF falls between a relatively
narrow range--then the new evidence will have less influence on the
new distribution. Figure 3 illustrates a typical example of the
relationship between the original and updated probability
distributions.
Figure 3: Hypothetical Example of the Relationship between Original
and Updated Probability Distributions:
[Refer to PDF for image: graph illustration]
Depicted are the following:
Original manufacturer's MTBF;
Bayesian update MBTF;
plotted against hours of operation.
Source: GAO analysis of NASA data.
[End of figure]
According to NASA representatives, the Bayesian estimation process was
found to be the most appropriate for ISS given:
* the Bayesian process is an accepted method to update data within the
probabilistic risk assessment discipline;
* the process incorporates original reliability data with observed
performance data; and:
* the Bayesian process can be performed even with no observed random
failures, which is important because many types of ORU have
experienced no failures over the life of the ISS.[Footnote 13]
Statistics textbooks and other authoritative sources of guidance
describe the use of the Bayesian estimation process to update
statistical variables, including MTBF. Moreover, the Bayesian
estimation process is a key element of NASA's recommended approach for
conducting probabilistic risk assessments, a basic requirement for all
major NASA programs and projects.[Footnote 14] NASA's probabilistic
risk assessment guidance describes the importance of the Bayesian
estimation process, and provides broad implementing guidance for NASA
program managers on incorporating Bayesian estimation into these
assessments. According to representatives from NASA's Office of Safety
and Mission Assurance--the organization within NASA that developed
NASA's probabilistic risk assessment guidance--the use of Bayesian
estimation is critical to ensuring that a program's probabilistic risk
assessment is supported by the best available evidence of system
reliability and performance. NASA officials indicated that Bayesian
estimation has proven to be more accurate than original manufacturers'
reliability estimates in predicting overall failure rates for ORUs on
the ISS. Bayesian estimation has been used to update MTBFs for
components on numerous NASA programs, including the space shuttle, the
Hubble Space Telescope, and the TDRSS family of satellites.
Similar methodologies are also used within government and industry to
update values for statistical parameters:
* The Nuclear Regulatory Commission employs the Bayesian estimation
process to update the failure rates of components in nuclear power
plants.
* The oil and gas industry employs the Bayesian estimation process to
update the failure rates of pressure vessels and other types of
equipment at industrial oil and gas facilities.
* The Food and Drug Administration has developed guidance for the
assessment of medical device effectiveness based on the Bayesian
estimation process.
While NASA's Office of Safety and Mission Assurance provides NASA
programs with broad implementing guidance on employing the Bayesian
estimation process, it leaves the details of implementing this process
to the discretion of the individual programs. The ISS program's
application of the Bayesian estimation process requires four key
sources of input data: (1) an ORU's original manufacturer-estimated
MTBF; (2) a parameter that governs the variance of the original MTBF's
probability distribution--a measure of the dispersion of values about
the MTBF's most likely value; (3) the total aggregated hours of
operation for the ORU population under consideration;[Footnote 15] and
(4) the actual number of random failures experienced by the population
of ORUs.
For three of the four sources of input data, the ISS program generally
has complete data readily available for each set of ORUs. For example,
each ORU's original estimated MTBF was obtained from the reliability
analysis performed by the original equipment manufacturer and
incorporated into an authoritative ISS database.[Footnote 16] In
addition, detailed data on each ORU's total hours of operation and the
number of random failures is maintained by the ISS program's logistics
and maintenance team, and is therefore readily available as a source
of reliable input data.
For the fourth source of input data--the parameter that governs the
variance of the original MTBF's probability distribution--there is no
readily available source. In most cases, originally predicted MTBF
estimates were provided to the ISS program from original equipment
manufacturers as a single value, without any estimate of variance.
Therefore, an assumption about the variance of MTBF must be made.
However, selection of a parameter that defines the variance of the
original MTBF's probability distribution is the most challenging
aspect of applying the Bayesian estimation process. The ISS tiger team
consulted at least two sources of information including a 1990 study
of external maintenance activities for Space Station Freedom--the
precursor to ISS--which had recommended using a parameter that
produced a relatively large variance for the original estimated MTBF,
and a 2005 independent technical consulting report by NASA's
Engineering and Safety Center (NESC), which recommended using a
parameter that would produce a relatively small variance.[Footnote 17]
After deliberating, the tiger team recommended adopting a variance
parameter for the original MTBF that would grant approximately "equal
weight" to an ORU's original estimated MTBF and its on-orbit
experience. The ISS program applies the Bayesian process to all ORUs
that have experienced random failures. For an ORU that has not
experienced any random failures, the ISS program allows each set of
ORUs to operate for at least one-half of the number of hours that is
equal to the original predicted MTBF for that set of units before
updating the MTBF by applying the Bayesian estimation process.
The ISS program believes that its choice of variance parameter is
appropriate and consistent with on-orbit experience to date. Using a
parameter that produces a larger variance would place more weight on
on-orbit experience. Because many ORUs have experienced no failures,
however, this could lead to overly optimistic updates of MTBF. On the
other hand, using a parameter that produces a smaller variance would
place more weight on the original estimated MTBF and lead to continued
unwarranted conservatism, according to the ISS program. Given the
current state of knowledge about ORUs, NASA's choice of an MTBF
variance parameter appears to be reasonable and based on sound
professional judgment. Thus far, NASA has not revisited the relative
weight given manufacturers' original reliability estimates versus on-
orbit experience, and currently has indicated no plans to do so.
However, as time goes on, the variance parameter selected could prove
to be overly conservative given that NASA has been seeing failure
rates for ORUs that are far more in line with Bayesian updates than
manufacturers' predictions. Continuing to weigh manufacturers'
original estimates equally with on-orbit experience through the ISS
end of life for those ORUs that have experienced a random failure
could lead to the purchase of an excess of spares because NASA could
continue to predict more failures than actually occur.
ISS program representatives have noted that the MTBFs based on
implementation of their Bayesian estimation process--with their choice
of MTBF variance parameter--are much closer to actual overall ORU
failure rates. For example, in 2005, a comparison of actual and
predicted failures of the external ISS ORU population showed the
hardware experiencing only about one-third the predicted number of
failures based on original MTBF estimates. However, given its
selection of a parameter for MTBF variance, the operational MTBF
calculated using ISS's Bayesian estimation process is sensitive to
single failures--that is, a single failure of an ORU can dramatically
change the operational MTBF calculated by using this process. In one
sense, this Bayesian update process does exactly what one would
expect--modify the original MTBF based on observed failure data.
Because many ISS ORUs have operated for well beyond their original
predicted MTBF without experiencing a random failure, and consequently
have updated operational MTBFs that are several times greater than
their original predicted MTBFs, a single random failure will quickly
reverse this growth. For example, successive annual updates of the
operational MTBF (in hours) for the ISS pump module assembly increased
each year through 2009. A single failure of the pump module assembly
in July 2010,[Footnote 18] however, resulted in a significant decrease
in MTBF--from an October 2009 estimate of 104,683 hours to an August
2010 estimate of 58,119 hours. This decrease of 44 percent could
result in the need to procure additional spares. Table 1 shows how the
estimated MTBF of the pump module assembly has changed over time.
Table 1: Successive Annual Updates of MTBF (in Hours) for the ISS Pump
Module Assembly:
Pump module assembly: Hours;
Original MTBF in October 2006 data: 39,800;
Updated MTBF October 2007: 52,450;
Updated MTBF October 2008: 86,560;
Current MTBF October 2009: 104,683;
Estimated runtime hours at failure per ISS risk team to August 3,
2010: 63,600;
New MTBF Bayesian updated with one failure August 2010: 58,119.
Source: NASA.
[End of table]
Similarly, the ISS television camera interface converter has not yet
experienced any random failures. Consequently, its predicted MTBF has
grown by more than 170 percent--from an original value of 100,675
hours to 272,908 hours (as of October 2010). A single random failure
of this device today, however, would cut the value of the operational
MTBF by more than half its current value--to about 134,000 hours.
These large changes in MTBF may have consequences for the results of
the ISS functional availability risk assessment, and could result in
the need to procure additional spares as discussed below. Given the
fact that the majority of ORUs on the ISS have yet to experience a
random failure, this issue is one that could affect sparing decisions
and place the program at risk of procuring too few spares. NASA
officials indicated that the agency mitigates this risk by updating
its sparing estimates annually.
Process for Establishing Functionality Targets and Confidence Goals Is
Reasonable but Rationale for Decisions Is Not Well Documented:
NASA has reasonable processes in place for establishing functionality
targets and confidence goals for each ISS function being modeled in
its functional availability risk assessment.[Footnote 19] The ISS
program manager considers the recommendations of system experts who
validate each functionality target to ensure that it meets all science
utilization and crew habitability requirements. The ISS program
manager's rationale for establishing functionality targets and
confidence goals is also driven by an assessment of the operational
needs of ISS as a whole, and by whether the needs satisfied by a given
function might be accomplished in some other way. NASA officials
indicated, for example, that the program manager's decision to set the
functionality target for S-band communications and tracking as "at
least one of two strings operational" instead of "both strings
operational" was influenced by the planned upgrades to ISS Ku-band
capability, which will be able to perform some functions that could
previously be performed only by using S-band, thereby mitigating the
loss of S-band redundancy. Finally, ISS functionality targets and
confidence goals are revisited based upon the results of each annual
functional risk assessment. Table 2 below shows several examples of
the 27 ISS functions NASA includes in its functional availability risk
assessment.
Table 2: Examples of ISS Functions Included in a Functional
Availability Risk Assessment:
Function: Command & Data Handling;
Purpose: Provides command and control capability of ISS via a network
of computers, data buses, and software.
Function: Electrical Power System;
Purpose: Generates, stores, and distributes electrical power.
Function: Structures & Mechanisms;
Purpose: Provides the primary and secondary structures of the ISS,
such as trusses and payload racks, as well as ISS mechanisms, such as
the joints and gimbals for positioning solar arrays.
Function: Thermal Control System;
Purpose: Maintains temperature control of the ISS.
Source: NASA.
[End of table]
Once initial functionality targets and confidence goals have been
established for each of the 27 functions, the ISS program employs a
Monte Carlo simulation to predict how many spare ORUs would be
required to meet these targets and confidence goals through the ISS
end of life. The ISS program also conducts a sensitivity analysis to
determine how the number of required spares would change if the
functionality targets and confidence goals were relaxed. For example,
the sensitivity analysis may reveal that a 98 percent confidence goal
for a given functionality target can be met with a given quantity of
spares, while a 99 percent confidence goal would require two to three
times as many spares. Similarly, analysis may reveal that while one
functionality target (e.g., three of four strings operational) may be
achievable with a certain number of spares, achieving a more demanding
functionality target (e.g., four out of four strings operational)
might require several times as many spares. This result would not
necessarily support the less demanding functionality target and
confidence goal in all cases--despite the significant increase in the
number of spares required--since the 99 percent confidence goal may be
justified based on safety-of-life concerns. This assessment, however,
allows the ISS program to see the costs, in terms of the required
number of spares, of setting functionality targets and confidence
goals at particular levels.
On the basis of this analysis, a technical community of system experts
supporting each modeled function validates each functionality target
and confidence goal and recommends them to the ISS program manager.
The ISS program manager is free to reject these recommendations based
on his or her broader knowledge of all ISS functions and other factors
affecting sparing decisions. For example, during the 2010 functional
availability risk assessment,[Footnote 20] the technical community for
the S-band communications and tracking function recommended a
functionality target of at least one of two strings operational, and a
confidence goal of 99.5 percent for maintaining this target. The
program manager agreed with the functionality target, but decided to
set the confidence goal at 98 percent. By setting a lower confidence
goal, the number of additional spares required dropped from a total of
six for all three types of ORUs to just two.
According to the ISS program manager and the system experts who make
the functionality target recommendations, they go through a
deliberative process wherein options for functionality targets and
confidence goals are developed and presented to the ISS program
manager at the Space Station Program Control Board meeting. These
presentations contain the technical community's recommendation on the
most appropriate functionality target and confidence level for a given
function. As illustrated above, the program manager has the discretion
and authority to select functionality targets and confidence levels
other than the ones recommended by the technical community. Agency
officials indicated that while the program manager's rationale for not
accepting the recommendation of the technical community is discussed
at these meetings, the meeting minutes do not typically capture the
rationale for the program manager's decision to deviate from the
recommendation of the system experts. Our Standards for Internal
Control in the Federal Government[Footnote 21] require, however, that
important events and transactions are clearly documented and readily
available for examination. This type of documentation provides
evidence for the benefit of future decision-makers to continue to
ensure that the rationale remains valid.
NASA's ISS Sparing Plan Relies on International and Domestic Vehicles
and Currently Includes Some Cargo Shortfalls:
NASA plans to use both international and domestic commercial launch
vehicles to ensure ISS spares are available through 2020 because these
are NASA's only near-term options for resupplying the ISS. Planned
launches, however, may not provide enough cargo capability to cover
anticipated growth in scientific utilization or margin for unforeseen
ISS maintenance needs, especially from 2015 through 2020.
Additionally, delays in domestic commercial launch vehicle development
could affect sparing deliveries planned for 2012 and beyond.
NASA plans to rely on ISS international partner and new commercial
launch vehicles to meet crew needs as well as support maintenance and
scientific experiments through 2020. The agency has agreements in
place with the European and Japanese space agencies for their
respective vehicles, the ATV and HTV, to conduct sparing missions
beginning in 2012. The ATV and HTV are unmanned vehicles, which have
flown to the ISS, and carry such items as hardware and water.[Footnote
22] NASA expects to employ a total of 13 international partner
launches--7 from 2012 to 2015 and 6 from 2016 through 2020.
As we have reported, the international partner vehicles alone cannot
meet ISS sparing needs.[Footnote 23] Consequently, NASA plans to use
two types of domestic commercial launch vehicles to maintain ISS from
2012 through 2020. Development of these vehicles--the Falcon 9 and
Taurus II--was fostered under a NASA-initiated effort known as
Commercial Orbital Transportation Services (COTS). These vehicles are
being developed by private industry corporations SpaceX (Falcon 9) and
Orbital Sciences (Taurus II). In late 2008, NASA awarded contracts to
both companies to provide cargo transport services to the ISS. Only
SpaceX's Falcon 9 vehicle is designed to safely return significant
amounts of cargo to earth, such as scientific experiments. NASA
anticipates that SpaceX will begin providing that capability in 2012.
Commercial vehicles are essential to sustaining and utilizing the ISS.
As Table 3 below indicates, SpaceX and Orbital are scheduled to fly 19
(73 percent) of the 26 launches NASA plans through 2015 and follow-on
SpaceX and Orbital commercial vehicles are expected to fly 24 (80
percent) of the 30 launches from 2016 through 2020.[Footnote 24]
Additional international partner flights from 2014 to 2020 beyond
those already scheduled are an unlikely option, according to NASA
officials, because the European and Japanese space consortium's
manufacturing facilities are not equipped to manufacture vehicles at a
rate faster than their current commitments.
Table 3: NASA's Planned Vehicle Launches for 2012 to 2020 to Resupply
the ISS:
Vehicles: ATV;
2012: 1;
2013: 1;
2014: 1;
2015: [Empty];
2016: 1;
2017: [Empty];
2018: [Empty];
2019: 1;
2020: [Empty];
Total: 5.
Vehicles: HTV;
2012: 1;
2013: 1;
2014: 1;
2015: 1;
2016: 1;
2017: 1;
2018: 1;
2019: [Empty];
2020: 1;
Total: 8.
Vehicles: SpaceX;
2012: 3;
2013: 3;
2014: 3;
2015: 2;
2016: [Empty];
2017: [Empty];
2018: [Empty];
2019: [Empty];
2020: [Empty];
Total: 11.
Vehicles: Orbital;
2012: 2;
2013: 2;
2014: 2;
2015: 2;
2016: [Empty];
2017: [Empty];
2018: [Empty];
2019: [Empty];
2020: [Empty];
Total: 8.
Vehicles: SpaceX Follow On Vehicle A;
2012: [Empty];
2013: [Empty];
2014: [Empty];
2015: [Empty];
2016: 3;
2017: 3;
2018: 2;
2019: 3;
2020: 3;
Total: 14.
Vehicles: Orbital Follow On Vehicle B;
2012: [Empty];
2013: [Empty];
2014: [Empty];
2015: [Empty];
2016: 1;
2017: 2;
2018: 3;
2019: 2;
2020: 2;
Total: 10.
Vehicles: Total;
2012: 7;
2013: 7;
2014: 7;
2015: 5;
2016: 6;
2017: 6;
2018: 6;
2019: 6;
2020: 6;
Total: 56.
Source: GAO analysis based on NASA documents.
Note: This table does not include flights by the Russian Soyuz or
Progress vehicles.
[End of table]
Although NASA expects domestic commercial launch vehicles to deliver
the bulk of cargo required by the ISS through 2020, NASA strategic
planning manifests indicate that NASA may not have sufficient
capability to carry all the cargo that could be needed on the ISS. The
manifests show that, when anticipated growth in national laboratory
demands and margin for unforeseen maintenance needs are accounted for,
the current number of flights NASA is planning for may not cover all
of NASA's anticipated needs beginning in 2014.[Footnote 25] These
shortfalls amount to a total of 2.3 metric tons--approximately the
cargo that one SpaceX commercial vehicle can transport to the ISS.
NASA has indicated, however, that these shortfalls could increase or
decrease because transportation demand, vehicle flight rate, and
vehicle capability projections beyond 2015 are estimates only.
Additionally, ISS agreements with commercial vehicle providers are not
in place beyond 2015 and agreements for use of international partner
vehicles beyond 2015 are in negotiations.
Figure 4: Cargo Capability of U.S. Commercial and International
Partner Vehicles Versus NASA's ISS Sparing Needs from 2012 to 2020:
[Refer to PDF for image: combined vertical bar and stacked line graph]
Calendar year: 2012;
Capacity:
Domestic/commercial: 9,693;
International partner: 6,337;
Demand:
Other cargo: 5,613;
National lab growth: 0;
Contingency maintenance: 0.
Calendar year: 2013;
Capacity:
Domestic/commercial: 10,819;
International partner: 7,115;
Demand:
Other cargo: 13,289;
National lab growth: 1,192;
Contingency maintenance: 3,453.
Calendar year: 2014;
Capacity:
Domestic/commercial: 10,819;
International partner: 7,129;
Demand:
Other cargo: 14,751;
National lab growth: 2,722;
Contingency maintenance: 759.
Calendar year: 2015;
Capacity:
Domestic/commercial: 8,479;
International partner: 3,868;
Demand:
Other cargo: 11,484;
National lab growth: 2,692;
Contingency maintenance: 915.
Calendar year: 2016;
Capacity:
Domestic/commercial: 8,919;
Demand:
International partner: 7,509;
Demand:
Other cargo: 12,451;
National lab growth: 2,472;
Contingency maintenance: 1,210.
Calendar year: 2017;
Capacity:
Domestic/commercial: 10,819;
International partner: 4,057;
Demand:
Other cargo: 11,849;
National lab growth: 2,628;
Contingency maintenance: 1,210.
Calendar year: 2018;
Capacity:
Domestic/commercial: 10,379;
International partner: 3,845;
Demand:
Other cargo: 12,499;
National lab growth: 3,017;
Contingency maintenance: 1,210.
Calendar year: 2019;
Capacity:
Domestic/commercial: 10,819;
International partner: 3,452;
Demand:
Other cargo: 12,378;
National lab growth: 2,628;
Contingency maintenance: 1,210.
Calendar year: 2020;
Capacity:
Domestic/commercial: 1,0819;
International partner: 3,845;
Demand:
Other cargo: 12,413;
National lab growth: 3,017;
Contingency maintenance: 1,210.
Source: GAO analysis of NASA data.
[End of figure]
As we have previously reported, both SpaceX and Orbital are working
under aggressive schedules and have experienced delays in completing
demonstrations.[Footnote 26] SpaceX may gain some ground in its
schedule by combining demonstration missions planned for 2011, but
this likely would not provide much overall schedule gain because
previous delays had already placed the program about 18 months behind
schedule. Additionally, in the summer of 2011 Orbital experienced an
engine failure during testing, which could affect its planned launch
vehicle availability. NASA has made efforts to accommodate delayed
commercial vehicle development, including use of the final shuttle
flight in July 2011 to pre-position additional ISS spares. However, if
the commercial vehicle launches do not occur as planned in 2012, the
ISS could lose some ability to function and sustain research efforts
due to a lack of alternative launch vehicles to support the ISS. The
need for these vehicles to launch in late 2012 could become even more
important as a result of the failure of the Russian Progress re-supply
mission in August 2011,[Footnote 27] one of the last Progress flights
that NASA had contracted with Russia to transport cargo to the ISS. As
noted by NASA officials, the Russian vehicle was to provide supplies
that would help sustain crew and scientific experiments aboard the
ISS. At this time, NASA officials indicated that the agency does not
believe the loss of the Progress flight increased the need for the
commercial launches in 2012 because the agency anticipates that a
Progress flight in 2013 will carry the supplies to the ISS and provide
margin against delays in the commercial vehicle launches. However, if
that Progress flight is not conducted, NASA could become more reliant
on the commercial vehicles.
NASA Has Budgeted for Estimated Hardware Spares through 2020:
To ensure that spares essential to ISS operations will be available,
NASA has budgeted approximately $900 million to supply ISS hardware
spares needed to meet functional targets through 2020. NASA officials
indicated that, of that amount, the agency plans to spend about $515
million procuring ISS spares to be used from 2010 through 2015 and
about $384 million for spares to be used from 2016 through 2020.
According to NASA officials, the difference in budgets between the
2010 through 2015 and 2016 through 2020 time frames reflects the need
to procure fewer spares from 2016 through 2020 because NASA already
purchased and pre-positioned spares aboard the ISS. The officials
indicated that this strategy allowed for schedule margin against
delayed development of commercial launch vehicles; that is, spares
would already be pre-positioned on the ISS in case commercial launch
vehicles were not available as planned in 2012. Additionally, the
officials stated that the strategy of graceful degradation reduces the
amount of spares required as the ISS nears its planned end of life in
2020.
As shown in Table 4 below, of the $515 million planned for spares to
be used through 2015, NASA officials anticipate that the agency will
spend about $288 million to procure spares for the Environmental
Control and Life Support System.[Footnote 28] The agency officials
expect NASA to use the remaining funds, about $227 million, to procure
spares for the electrical power system; the thermal control system;
communications and tracking systems; command and data handling; and
structures and mechanisms, such as hatches and payload racks.
Table 4: Cost Estimates for ISS Hardware Spares to be Used through
2015:
System type: Electrical Power System;
Total cost: $31.6 million;
Last delivery date: 2012.
System type: Structures and Mechanisms;
Total cost: $92.8 million;
Last delivery date: 2012.
System type: Command and Data Handling;
Total cost: $1.3 million;
Last delivery date: 2009.
System type: Communications and Tracking;
Total cost: $76.5 million;
Last delivery date: 2013.
System type: Thermal Control System;
Total cost: $24.5 million;
Last delivery date: 2012.
System type: Regenerative ECLSS;
Total cost: $233.0 million;
Last delivery date: 2013.
System type: ECLSS;
Total cost: $55.2 million;
Last delivery date: 2012.
System type: Total all systems;
Total cost: $514.9 million.
Source: GAO analysis based on cost data provided by NASA officials.
[End of table]
Of the $384 million NASA plans to use to procure spares for use in
2016 through 2020, agency officials estimate NASA will need about $196
million to replace ISS batteries and lighting and about $115 million
for spares for electrical power, environmental control and life
support, and other systems. The remaining budget estimate of about $73
million, as noted by NASA officials, is expected to be used for repair
of spares returned to earth via SpaceX commercial launch vehicles. The
officials emphasized that budget estimates and hardware quantities for
2016 through 2020 are not firm, and they continually refine both
estimates and quantities of spares needed as the agency gains
increased knowledge of how long hardware spares will last. For
example, officials indicated that they are already adjusting budget
projections based on the results of the 2011 functional availability
assessment, which indicated that some functions may need more spares
than anticipated while other functions may require fewer spares.
NASA Is Using Reasonable Methods to Assess Structures and Hardware:
NASA is using reasonable analytical methods to assess structural
health as well as station hardware and ORUs that could affect the
safety of the station. NASA's software tools, in use since the
program's inception, indicate the ISS is structurally sound to support
continued operations through 2020 and potentially beyond with some
mitigation. Other federal agencies recognize analytical assessments as
valid methods for evaluating structural health, and the software NASA
uses is widely used by industry. Currently, however, NASA has assessed
only 40 percent, by weight, of the assembled ISS because most of the
ISS structures have not been on orbit long enough to accumulate the
data needed for analysis.[Footnote 29] NASA expects to complete ISS
structural assessments in early 2016. NASA plans to use improved
software tools for evaluating life extension of ISS's structures and
expects these tools to provide greater insight into the structural
health of the ISS. By 2012, NASA expects to complete analytical safety
and mission assurance reviews designed to assess operations of the ISS
as a whole as well as identify ORUs and other hardware failure of
which could result in an increased risk of loss of station or loss of
crew. Through 2015, NASA plans to develop methods to mitigate issues
identified. When these mitigation actions are developed and
implemented, NASA expects to have increased assurance that the ISS can
safely operate through 2020 and potentially beyond.
Analysis Ongoing but NASA Expects the ISS to Remain Structurally Sound
with Mitigation:
Based on prior analysis of structural life usage through 2015 and the
robust design of the ISS primary structures, NASA anticipates that--
with some mitigation--the ISS will remain structurally sound to
support operations through 2020, though its analysis through 2020 is
still ongoing. NASA standards for manned spaceflight systems require
the ISS program to monitor the structural health of the ISS.[Footnote
30] Other federal agencies--including the Federal Highway
Administration, Federal Aviation Administration, and National
Institute of Standards and Technology--recognize visual, physical,
radiographic (x-ray), and sonographic inspections as well as
analytical assessments as valid methods for evaluating the structural
health of bridges or airplanes. Visual inspections of the ISS are
limited to using television cameras to assess the station's exterior,
but the cameras do not provide sufficient detail for structural
inspection. Additionally, the position of payload racks within the ISS
severely hampers visual inspection of the station's interior weld
seams. Physical assessments are limited to one ISS structure--the US-
funded, Russian-built Functional Cargo Block.[Footnote 31] These
ground-based assessments, conducted on a full-scale model of the cargo
block, have shown that it should remain structurally sound through
2028.[Footnote 32] According to agency officials, NASA has not
attempted to develop techniques or equipment to conduct x-ray or
sonographic inspection of the ISS. The officials stated that doing so
would be extremely expensive, impractical, or both.
Because NASA cannot feasibly use traditional methods to assess the
majority of the ISS on orbit, the agency relies on analytical
assessments to determine ISS structural health. Since program
inception, NASA has used various versions of a software tool known as
NASGRO® [Footnote 33] as its key analytical tool for monitoring the
structural health of the U.S. portions of the ISS.[Footnote 34] NASGRO
assumes that non-discernible cracks are present in any given structure
and predicts the growth rate and likely size of a crack or "crack
growth" based on: (1) the shape of the structure, (2) the material
composition of the structure, and (3) the physical loads placed on the
structure. NASGRO is widely used by government and industry including
the Department of Defense (DOD), Federal Aviation Administration, The
Boeing Company, Lockheed Martin, Honda, Airbus, and Spirit
Aerosystems. NASGRO is also used by the International Partners, i.e.,
the Japanese and European space agencies to monitor the structural
health of their elements. NASA, in conjunction with other users,
continually provides input for the update and release of new versions
of NASGRO with increased analysis capabilities and materials
knowledge. One NASGRO industry user we talked to said that his company
chose to rely on NASGRO as its tool for predicting structural health
because it is conservative in its projections, has an extensive
materials properties database, and is a Federal Aviation
Administration-accepted analysis method for fatigue and damage
assessment of structures.
NASA's approach to assessing structural health compares initial NASGRO
predictions of crack growth with updated NASGRO predictions of crack
growth. NASA used NASGRO to calculate initial predictions of crack
growth during the ISS design phase based on the mechanical, thermal
and pressure loads NASA expected the ISS to experience throughout its
operational life ending in 2015. NASA annually updates crack growth
predictions based on the actual loads the ISS has experienced since
being placed in orbit.[Footnote 35] NASA prepares the updated
predictions of crack growth with the same version of NASGRO used to
prepare the initial prediction. NASA prepared the initial predictions
of crack growth for each structural component with the version of
NASGRO that was current at that time. As the ISS was developed over
several years, NASA uses several versions of NASGRO to assess
structural health.
For these comparisons to be useful, the structural elements of the ISS
under examination must have been in orbit long enough to accrue
meaningful differences between predicted and actual loads. Thus far,
NASA has found that the actual loads experienced by ISS structures
have been generally less stressful than those originally predicted.
The last completed assessment, in fiscal year 2010, examined the ISS
structures placed in orbit as of November 2002, which represents
approximately 40 percent of the ISS measured by weight.
The structural health assessment uses a risk-based screening approach
and examines in detail only those components originally estimated to
have less than 6 service lives or safety margins of less than 10
percent.[Footnote 36] According to program officials, because ISS
developers were conservative in their design approaches and in most
cases built hardware that exceeded requirements, most structural
locations do not need to be examined in detail. As of the last
analysis, completed in November 2010, of the 10,000+ structural
locations within the on-orbit segments of the ISS, 829 locations were
examined in detail because they met screening criteria. Of the 829
locations examined in detail, 196 are using structural life faster
than originally predicted, but only 19 no longer meet the original
requirement of 60 years of service life. Fifteen of the 19 are located
on the P6 truss.
Figure 5: P6 Truss in Interim and Permanent Locations:
[Refer to PDF for image: 2 photographs]
Interim location of P6 truss;
Permanent location of P6 truss.
Source: NASA.
[End of figure]
The P6 truss--an end portion of the truss that serves as the
structural backbone of the ISS, as shown in figure 5 above--holds 4 of
the ISS's 16 solar panels. The P-6 truss provides an example of the
impact of actual loads data on the determination of structural health
of ISS structures. The truss was left in what was planned as an
interim location for an extended period of time following the Space
Shuttle Columbia accident. This location left it exposed to greater
extremes of hot and cold temperatures than anticipated. According to
program officials, the P6 truss may require mitigation to remain sound
through 2020 and potentially beyond. NASA is evaluating mitigation
processes, such as adding thermal insulation blankets and changing how
and where vehicles dock to the ISS, to slow down the rate at which the
locations on the P6 truss, as well as the other locations, are
expending structural life.
As part of its life extension effort, NASA is using a single version
of NASGRO--NASGRO 6.0--to develop new crack growth baselines for
assessing structural health through 2020 and potentially beyond. The
agency is deriving the new baselines from the actual on-orbit loads
the ISS has experienced since being placed in orbit and predicted
loads for using the ISS through 2028.[Footnote 37] The agency expects
to incorporate these new baselines into biennial ISS structural health
assessments beginning in 2012. Methodologically, these assessments
will be virtually identical to the earlier assessments, but they will
use the new baselines and improved NASGRO software with increased
capabilities. NASA plans to establish these new baselines in 3 phases,
with the last one established in 2015.
* Phase 1, beginning in 2009 and ongoing through 2012, examines
structures with original certifications expiring on or before 2013.
* Phase 2, beginning in 2012 and ongoing through 2014, examines
structures with original certifications expiring on or before 2017.
* Phase 3, beginning in 2012 and ongoing through 2016, examines
structures with original certifications extending beyond 2020.
On the basis of the results of these analyses, NASA will determine
what mitigation actions need to be put in place to ensure the ISS
remains structurally sound through 2020.
On the basis of familiarity with NASGRO software, NASA officials
expect the updated software to provide greater insight into the
structural health of the ISS. For example, NASA expects the new
version of NASGRO to better capture the impact of high cycle, low
impact loads, such as those caused by astronauts running on the
treadmill on the structural health of the ISS. Further, NASA expects
the new software to support more in-depth analysis of how thermal,
pressure and mechanical loads affect ISS structures. NASA also
anticipates that improved knowledge of materials properties data
within the software will support more informed analysis of how the
structures of the ISS respond to different loads and environments.
When NASA begins using the updated software tools, part of the
agency's input, the detailed design of the structures, will be based
on existing models of the ISS. Officials indicated, however, that in
some instances, these models are not detailed enough to take advantage
of all the expanded features of the updated NASGRO software. According
to the officials, these models were designed to be used with less
sophisticated software than NASGRO 6.0. In areas not expected to meet
service-life requirements, one of the mitigation strategies available
is refined modeling with increased details. For example, more detailed
models could allow NASA to more precisely pinpoint structural issues
and focus mitigations accordingly.
NASA Is Employing Reasonable Methodologies to Ensure Safety of ISS
Hardware through 2020:
NASA is employing reasonable analytical methodologies to assess the
extent to which ISS hardware, including ORUs and associated components
such as flex hoses and tubing, can be safely operated through 2020 and
potentially beyond. These methodologies employ established safety and
mission assurance practices, governed by agency and program
directives,[Footnote 38] to perform a safety and mission assurance
assessment as well as rely on program experts to assess the safety of
hardware whose failure could result in the loss of ISS crew or the ISS
as a whole.
NASA is performing two types of safety assessments. The first, a
safety and mission assurance assessment, is an analytical review of
hazards to the ISS as a whole providing a "top-down" look at how
extending the life of the ISS through 2020 and potentially beyond
affects ISS safety. Additionally, for the second assessment, NASA is
using program experts to assess the ORUs[Footnote 39] and other
hardware, whose failure could result in loss of ISS crew or the entire
ISS. This assessment is a "bottom-up" analytical review to determine
whether this hardware can be safely operated through 2020 and
potentially beyond. According to NASA officials, addressing safety
from two different perspectives--top down and bottom up--offers
increased assurance that safety issues are fully addressed.
NASA's safety and mission assurance practices call for identification
of safety hazards and mitigation planning to address hazards. Safety
hazards on the ISS are varied and consist of hazards to both the crew
and the station. Examples of hazards include loss of crewmembers due
to inability to reenter the ISS after conducting a spacewalk or loss
of ISS attitude control. As indicated by NASA officials, the agency
originally documented these hazards in 92 hazard reports applicable to
ISS operations through 2015. NASA is now in the process of conducting
a "top-down" review of these reports to determine if existing hazards
require additional mitigation as a result of extending ISS life
through 2020 and potentially beyond. An official working on the
assessment stated that preliminary results of this assessment indicate
that 58 hazard reports are affected by extending ISS life beyond 2015
and require further study. Mitigation actions could take a variety of
forms, for example, procuring additional ORUs, updating hazard reports
and operational procedures, and performing hardware modifications and
redesigns.
In conjunction with the "top-down" safety and mission assurance
assessment, NASA is using existing hardware teams who are responsible
for and knowledgeable of their respective systems to assess the impact
of extending station life on ORUs and other hardware. These teams are
conducting in-depth, "bottom-up" reviews of ORUs and associated
hardware, such as hose assemblies and tubing, to identify hardware
that must be replaced before failure because the hardware's failure
could lead to loss of crew or station. According to NASA officials,
the analysis is currently in progress, but preliminary results
indicate that some hardware will not be affected by ISS life
extension, while other hardware may require replacement or other
mitigation to ensure it does not fail and result in a catastrophic
loss of life or station. For example, NASA's early findings show that
52 hardware items, including tubes and hose assemblies in the Internal
Thermal Control System, the system that moderates temperature
throughout the ISS, can be used until they fail because their failure
would result only in degradation of system functions and not endanger
the ISS or crew.
NASA expects to complete both the "top-down" safety and mission
assurance and "bottom-up" hardware assessments by 2012. Agency
officials stated that they will begin mitigation planning immediately
after the completion of both assessments and continue that planning
through 2015. They noted that the ISS program will implement
corrective actions as mitigation plans are developed and put in place,
so that some mitigation efforts will occur as early as 2012 and some
after 2015. When mitigation actions are developed and in place, NASA
expects to have increased assurance that the ISS can safely operate
through 2020 and potentially beyond.
Conclusion:
Extending the operational life of the ISS to 2020 will allow NASA and
the United States to capitalize on the nearly two decades and billions
of dollars invested in its construction and operation. During the time
required to finish ISS construction, however, many changes occurred in
the assumptions and conditions underlying NASA's initial ISS
supportability strategy--the planned operational life of the ISS has
extended, the space shuttle program has ended, and ORUs on the ISS are
lasting longer than originally expected. In response to the changing
conditions, NASA has appropriately evolved its strategy for supporting
the ISS through 2020. Assessing ISS extended life necessarily requires
the use of sophisticated analytical techniques and judgments. To that
end, the agency has adopted reasonable approaches and methodologies
for estimating the spares needed for continued operation and ensuring
that the ISS structure and hardware are safe for operation through
2020 and potentially beyond. For example, NASA actively incorporates
new knowledge to update estimates of MTBF for its ORUs, which provides
greater efficiency in the use of resources necessary for purchasing
spare parts. While this approach is sound, it is important that this
process continue to evolve as the agency gains knowledge to ensure the
most effective and efficient use of resources for ISS utilization
through 2020. NASA has not indicated, however, that it plans to
incorporate this new knowledge by revisiting the variance parameter
that governs the relative weight given to on-orbit experience versus
manufacturers' original reliability estimates in conducting Bayesian
updates. Doing so when the appropriate amount of knowledge is obtained
could allow NASA to achieve even greater efficiency in the purchase of
spares. In considering these assumptions, however, NASA must balance
efficiency with risk. If NASA weights on-orbit experience too heavily,
it risks overstating the MTBF and potentially underestimating the
number of spares needed, especially for ORUs that have not experienced
a random failure. Conversely, weighing manufacturers' original
estimates too heavily could result in NASA expending resources on
unneeded spares. Moving forward, NASA needs to carefully balance these
competing information sources based on future on-orbit experience to
ensure an appropriate risk posture in the procurement of spares. The
basis underlying NASA's functionality targets and goals, which are
revisited annually, could also be affected by factors such as
increased knowledge of performance and functionality and additional
capabilities provided as new launch vehicles become available.
Understanding the rationale behind NASA's functionality targets and
goals is important so that as new information is gained it can be
assessed to ensure that it remains valid. For example, if a particular
goal was set at 98 percent due to the assumed availability of spares
and those spares become unavailable, that goal may no longer be
achievable and mitigations or an alternative approach would need to be
devised to meet that goal. Without documentation of the rationale
behind the original decision, ISS officials have limited means to
assess whether the rationale remains valid and the goal is achievable.
Recommendations for Executive Action:
As the ISS program accumulates additional knowledge about the on-orbit
performance of ORUs, we recommend that the NASA Administrator direct
the ISS program manager to revisit, as appropriate, the relative
weight given to manufacturers' original reliability estimates and to
the actual reliability of ORUs based on on-orbit experience by
reexamining the program's choice of a value for the parameter that
governs the variance of the original MTBF's probability distribution.
To ensure decision makers have a full understanding of the rationale
behind ISS functionality targets and confidence levels to better
inform future decisions, we recommend that the NASA administrator
direct the ISS program manager to ensure these rationales are
documented, at a minimum, in the minutes from meetings of the Space
Station Program Control Board.
Agency Comments and Our Evaluation:
In written and oral comments on a draft of this report (see appendix
II), NASA concurred with our recommendations. NASA acknowledged the
benefit of revisiting the relative weight given to manufacturers'
original reliability estimates and to the actual reliability of
orbital replacement units and indicated that the ISS program office
intends to reexamine these weighting values as appropriate. NASA also
acknowledged that the ISS program office should document the rationale
underlying the Program Manager's decisions concerning ISS
functionality targets and confidence levels and indicated that the ISS
Program Manager is already in the process of implementing this
recommendation. Separately, NASA provided technical comments, which
have been addressed in the report, as appropriate.
We are sending copies of this report to the appropriate congressional
committees. We are also sending copies to NASA. This report will also
be available at no charge on the GAO website at [hyperlink,
http://www.gao.gov].
Should you or your staff have any questions concerning this report,
please contact me at (202) 512-4841 or chaplainc@gao.gov. Contact
points for our Offices of Congressional Relations and Public Affairs
may be found on the last page of this report. GAO staff who made key
contributions to this report are listed in appendix III.
Signed by:
Cristina T. Chaplain:
Director Acquisition and Sourcing Management:
[End of section]
Appendix I: Scope and Methodology:
We interviewed and obtained briefings and relevant documents from
knowledgeable National Aeronautics and Space Administration (NASA) and
contractor officials on the content and decision-making approach NASA
used to prepare its assessment in response to section 503 of the NASA
Authorization Act of 2010.[Footnote 40] directing the agency to assess
its plans for ensuring essential spares were available to support the
International Space Station (ISS) through the station's planned
operational life. We also reviewed the scope and methodology NASA used
in the ISS assessment. We conducted our work at NASA headquarters in
Washington, D.C., and NASA's Johnson Space Flight Center, in Houston,
Texas.
For purposes of assessing NASA's findings and approach to estimating
spares essential for ISS operation through 2020, we focused our
efforts on the statistical techniques NASA used to calculate
operational mean time between failure (MTBF) rates and the modeling
techniques NASA uses to assess functional availability as well as
NASA's process for establishing functionality targets and confidence
levels for ISS functional systems. For a limited number of orbital
replacement units (ORU), we recalculated the values NASA obtained for
the operational MTBF based on its statistical methodology. We also
conducted limited tests of the data NASA used to support the
functional availability model. We discussed the ISS program's approach
with NASA experts outside the ISS program at NASA headquarters. We
discussed NASA's approach for ensuring data reliability and
configuration control of ISS models and analytical systems with
cognizant and responsible officials. We also obtained and reviewed a
briefing on the results of an internal review of ISS program models
and analytical systems. We limited the scope of our assessment to the
scope of NASA's report, i.e., the sparing necessary to support
critical functions as modeled in the ISS functional availability
assessment. We did not examine the sparing or launch vehicle needs of
the international partners or the sparing needs of functions not
included in the ISS functional availability assessment.
For purposes of assessing NASA 's findings and approach to monitoring
and assessing the structural health of the ISS, we interviewed and
obtained briefings and relevant documents from knowledgeable NASA and
contractor officials at the Johnson Space Center. We obtained and
reviewed the results of NASA's past assessments of ISS primary
structures to identify issues that could affect ISS operations through
2020. We obtained and reviewed Department of Defense and NASA guidance
documents and determined NASA's basis for establishing four service
lives as its baseline requirement for structural life. We discussed
NASA's basis for establishing six service lives as the cutoff point
for conducting ongoing structural health assessments with
knowledgeable agency and contractor officials. Through discussions
with NASA's NASGRO experts at Johnson Space Center; with NASGRO
experts responsible for managing the commercial side of the NASGRO
structural analysis software at Southwest Research Institute; and
commercial users of NASGRO, we obtained an understanding of NASA's
structural heath assessment software tools and models, how NASA uses
them to support its assessments of structural health, and NASA's plan
for incorporating improved software tools into future assessments. We
reviewed Federal Aviation Administration and Federal Highway
Administration policies guiding the inspection of aircraft and bridges
and compared them to NASA's approach to measuring structural health.
We identified NASA's plans to use improved software with increased
knowledge of material properties and enhanced software tools to gain
improved measurements of structural health. We discussed with
knowledgeable NASA officials the mitigation strategies NASA is
considering to address structural problems discovered by structural
health assessments.
To determine the viability of NASA's findings and plans regarding
transportability for and supportability of the ISS, we examined NASA's
plans for transporting cargo to support the ISS through 2020 and
examined ISS launch vehicle schedules from 2012 through 2020. We
discussed those schedules, including launch vehicle availability, with
NASA officials. We also reviewed NASA launch manifests for 2012
through 2020. We compared the launch schedules to the planned cargo
manifests and determined whether planned launch vehicles would be
capable of meeting ISS supportability needs. Additionally, we reviewed
prior GAO reports and testimonies addressing commercial domestic
launch vehicle development and NASA's management and program
challenges. To evaluate the status of NASA's efforts to estimate the
costs of ISS spares through 2020, we obtained documents providing
information about NASA's past and predicted ISS sparing costs. We
discussed these estimates with knowledgeable NASA officials to gain
insight into how the cost estimates were developed and how changes in
cargo needs and vehicle availability might affect the estimates.
To assess NASA's findings and plans for ensuring ISS safe operations
through 2020, we interviewed and obtained briefings from NASA
officials on their methodologies for assessing the safety of the ISS
as a whole as well as the hardware that could affect safety of the
crew or station through 2020. Specifically, we reviewed the
methodology NASA is using to perform top-down safety and mission
assurance assessments. We also reviewed the methodology NASA is using
to perform bottom-up assessments of ISS hardware to determine if there
are ISS hardware safety issues associated with extending ISS mission
life. Additionally, we assessed NASA's approach to reviewing ISS
safety in terms of the agency's and Johnson Space Center's internal
regulations and guidance. We also discussed NASA's approach with ISS
program and contractor officials. We interviewed and obtained
briefings from NASA officials on the results-to-date of ISS safety and
mission assurance and hardware assessments and discussed issues that
could affect the safety of ISS operations through 2020 as well as
NASA's plans for mitigating any such issues.
We conducted this performance audit from February 2011 to November
2011 in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit
to obtain sufficient, appropriate evidence to provide a reasonable
basis for our findings and conclusions based on our audit objectives.
We believe that the evidence obtained provides a reasonable basis for
our findings based on our audit objectives.
[End of section]
Appendix II: Comments from the National Aeronautics and Space
Administration:
National Aeronautics and Space Administration:
Headquarters:
Washington, DC 20546-0001:
December 1, 2011:
Reply to the Attn of: Human Exploration and Operations Mission
Directorate.
Ms. Cristina Chaplain:
Director:
Acquisition and Sourcing Management:
United States Government Accountability Office:
Washington, DC 20548:
Dear Ms. Chaplain:
The National Aeronautics and Space Administration (NASA) appreciates
the opportunity to review and comment on the Government Accountability
Office (GAO) draft report entitled, "International Space Station:
Approaches for Ensuring Utilization Through 2020 are Reasonable but
Should Be Revisited as NASA Gains More Knowledge of On-Orbit
Performance" (GAO-12-162). NASA considers the management of the
International Space Station to be an important issue and greatly
values the constructive information and insights shared by GAO during
the course of this effort. We further appreciate the extreme
professionalism demonstrated by your review team and the continued
open communication maintained between GAO and NASA.
In the draft report, GAO addresses two recommendations to the NASA
Administrator (see below). NASA's responses to these recommendations
immediately follow.
Recommendation 1: We recommend that the NASA Administrator direct the
ISS program manager to revisit, as appropriate, the relative weight
given to manufacturers' original reliability estimates and to the
actual reliability of orbital replacement units (ORUs) based on on-
orbit experience by reexamining the program's choice of a value for
the parameter that governs the variance of the original mean time
between failure (MTBE's) probability distribution.
Management's Response: NASA concurs with the following comment. The
revisiting of weights assigned to reliability estimates is already
performed within the responsibilities of the ISS Program staff.
Recommendation 2: To ensure decision makers have a full understanding
of the rationale behind ISS functionality targets and confidence
levels to better inform future decisions, we recommend that the NASA
Administrator direct the ISS program manager to ensure these
rationales are documented, at a minimum, in the minutes from meetings
of the Space Station Program Control Board.
Management's Response: NASA concurs with the GAO's recommendation. The
documentation of rationale for ISS Program decisions is within the
scope of responsibility of the ISS Program Manager, and this
recommendation is already being implemented.
Thank you for the opportunity to comment on this draft report. If you
have any questions or require additional information, please contact
Mark Uhran at (202) 358-2233.
Sincerely,
Signed by:
William H. Gerstenmaier:
Associate Administrator for Human Exploration and Operations:
[End of section]
Appendix III: GAO Contact and Staff Acknowledgments:
GAO Contact:
Cristina T. Chaplain, (202)512-4841 or chaplainc@gao.gov:
Acknowledgments:
In addition to the contact named above, Shelby S. Oakley, Assistant
Director; John Warren, Analyst-in-Charge; Ana Ivelisse Aviles; Andrea
Bivens; Tana Davis; Jay Tallon; Sonya Vartivarian; Laura Greifner;
Roxanna Sun; and Sylvia Schatz made key contributions to this report.
[End of section]
Footnotes:
[1] Pub. L. No. 111-267.
[2] GAO, International Space Station (ISS): Ongoing Assessments for
Life Extension Appear to be Supported, [hyperlink,
http://www.gao.gov/products/GAO-11-519R] (Washington, D.C.: Apr. 11,
2011).
[3] Four service lives is a NASA and Department of Defense standard
for spacecraft. According to agency officials, most components are
designed to operate beyond four service lives.
[4] The structures of the ISS encounter thermal, pressure and
mechanical loads. According to program officials, thermal loads are
developed due to temperature gradients across or through a structure
and/or by a mismatch in materials at a joint; pressure loads are
developed due to pressure differences on the inside versus the outside
of a structure, such as the atmosphere in a crew compartment versus
the vacuum of space; and mechanical loads are developed when a force
is applied to the structure, such as vehicle dockings and undockings.
[5] National Aeronautics and Space Administration Authorization Act of
2008, Pub. L. No. 110-422, § 601; National Aeronautics and Space
Administration Authorization Act of 2010, Pub. L. No. 111-267, § 503.
[6] In 2004, President George W. Bush established a new space
exploration policy-A Renewed Spirit of Discovery: The President's
Vision for U.S. Space Exploration (Vision)-which called for the
retirement of the space shuttle and development of a new family of
exploration systems to facilitate a return of humans to the moon and
eventual human spaceflight to Mars.
[7] Department of Defense, Military Handbook: Reliability Prediction
of Electronic Equipment, MIL-HDBK-217F (Washington, D.C.: Feb. 28,
1995).
[8] Bayesian analysis is a method of statistical inference (named for
English mathematician Thomas Bayes) that allows one to combine prior
information about a population parameter with evidence from
information contained in a sample to guide the statistical inference
process.
[9] The ISS Reliability and Maintainability Panel at Johnson Space
Center in Houston, Texas, provides support to all ISS hardware teams
at all NASA centers.
[10] S-band is a radio band used by NASA to communicate with the ISS.
[11] Monte Carlo modeling is used to approximate the probability
outcomes of multiple trials by generating random numbers. Monte Carlo
models are widely used throughout government and industry to calculate
approximate probabilities for complex events when the calculation of
exact probabilities would be difficult or impossible.
[12] A general description of a probability distribution is a function
that describes the probability of a random variable taking certain
values.
[13] A probabilistic risk assessment is a comprehensive, structured,
and logical analysis method aimed at identifying and assessing risks
in complex technological systems for the purpose of cost-effectively
improving their safety and performance in the face of uncertainties.
Probabilistic risk assessments assess risk metrics and associated
uncertainties relating to likelihood and severity of events adverse to
safety or mission. (NASA Procedural Requirements 8715.3C, NASA General
Safety Program Requirements).
[14] NASA Procedural Requirements 8705.5A, Technical Probabilistic
Risk Assessment (PRA) Procedures for Safety and Mission Success for
NASA Programs and Projects.
[15] Because the failure rate is constant, the Bayesian estimation
process does not care whether the on-orbit run time is a single ORU
operating for 100,000 hours, or 4 identical ORUs operating for 25,000
hours.
[16] The ISS program maintains an "original MTBF" data field in its
Modeling and Analysis Data Set (MADS)--the database it maintains on
key orbital replacement unit parameters--and uses the information from
this data field in its Bayesian estimation calculations. We discussed
with ISS program representatives the steps they had taken to ensure
that the MTBF data captured in MADS reflect the data from the
reliability analysis performed by the original equipment
manufacturers. We also confirmed for a small number of orbital
replacement units, that the MTBF data contained in MADS matched the
MTBF data from the original equipment manufacturers' reliability
analysis. However, because many of these reliability reports were
performed well over a decade ago, and many exist only in hard copy at
a long-term storage facility, we did not conduct detailed tests of
these underlying data.
[17] NASA, Space Station Freedom: External Maintenance Task Team Final
Report. (July 1990). (This study is more commonly referred to as the
Fisher-Price report, after its two principal authors.)
[18] According to ISS program office representatives, the July 2010
failure of the pump module assembly has been categorized as a random
failure. However, if further analysis and testing reveal that the unit
failed for some other reason--for example, a workmanship problem--then
the failure would be re-categorized, and the Bayesian estimation
calculation would be repeated without the failure.
[19] The functional availability risk assessment employs a Monte Carlo
simulation model to calculate the number of spares required to meet
the functionality targets and confidence goals for each function.
Monte Carlo modeling is widely accepted by industry and government as
appropriate for these types of simulations.
[20] The 2010 functional availability assessment had focused on the
number of spares required to meet the functionality target and
confidence goals through 2020. The 2011 functional availability risk
assessment addressed meeting functionality targets and confidence
goals beyond 2020. However, we did not receive information on the 2011
assessment until late in our review and, therefore, did not have time
to assess the results.
[21] GAO, Standards for Internal Control in the Federal Government,
[hyperlink, http://www.gao.gov/products/GAO/AIMD-00-21.3.1]
(Washington, D.C.: November 1999).
[22] In 2008 and 2009, the ATV and HTV vehicles respectively flew to
the ISS and docked at the station to demonstrate their capabilities.
In 2011, both vehicles again launched. These flights were the second
for both systems.
[23] GAO, NASA: Commercial Partners Are Making Progress, but Face
Aggressive Schedules to Demonstrate Critical Space Station Cargo
Transport Capabilities, [hyperlink,
http://www.gao.gov/products/GAO-09-618] (Washington, D.C.: June 16,
2009).
[24] NASA has awarded contacts to SpaceX and Orbital for cargo
resupply services to the ISS through 2015. Planned follow-on vehicles
are the vehicles NASA will use for flights beyond those currently on
contract.
[25] NASA selected a nonprofit organization to facilitate increased
use of the ISS on-orbit research capabilities by entities outside of
NASA.
[26] GAO, NASA: Commercial Partners Are Making Progress, but Face
Aggressive Schedules to Demonstrate Critical Space Station Cargo
Transport Capabilities, [hyperlink,
http://www.gao.gov/products/GAO-09-618] (Washington, D.C.: June 16,
2009).
[27] The Progress vehicle's mission failed when the third stage engine
failed during launch.
[28] The Environmental Control and Life Support System (ECLSS) makes
the ISS habitable for crew by providing or controlling oxygen supply,
fire detection and suppression, waste management and water supply. The
total of $288 million includes $233 million NASA has budgeted to
procure spares for the Regenerative ECLSS, which recycles liquid waste
into potable water and oxygen. In terms of estimating the needed
spares, the Regenerative ECLSS is the most high-risk functional
system, as several of its ORUs are failing faster than originally
anticipated.
[29] The timing of initial structural assessments of ISS components
was originally driven by programmatic judgment calls as to when
predicted and actual loads had diverged enough to provide a meaningful
assessment. The timing and order of current assessments are now driven
by the assembly sequence of the ISS and NASA's desire to complete the
initial assessment of each component before its 15-year structural
certification expires.
[30] NASA Technical Standard-5007, General Fracture Control
Requirements for Manned Spaceflight Systems (March 13, 2001); and NASA
Technical Standard-5003, Fracture Control Requirements for Payloads
Using the Space Shuttle (October 7, 1996) establish the requirements
for protection against fracture within spaceflight system structures.
Fractures, simply put, are cracks that form in structures as a result
of the loads they encounter. Fracture criticality deals with the
likelihood that a structural element will fail under the loads it
experiences. For a component to be considered fracture critical, its
potential failure must cause loss of life, loss of mission, or loss of
a manned spaceflight system. Fracture control ensures a paper trail of
fracture analysis and non-destructive testing/evaluation.
[31] The Functional Cargo Block (FGB) was the first portion of the ISS
put into orbit.
[32] The Russian Space Agency used repetitive physical tests of a full-
scale model of the FGB to initially certify the FGB for 15 years of
life on-orbit. The original certification performed by the Russian
Space Agency was set to expire in 2013. To assess the viability of
using the FGB an additional 15 years to 2028, NASA relied on the
Russian Space Agency to conduct further testing on the full-scale
model originally used to certify the FGB. In July 2011, NASA and the
Russian Space Agency recertified the FGB for 15 additional years of
service--until 2028.
[33] NASGRO is a suite of programs used to: analyze fatigue crack
growth and fracture, perform assessments of structural life, process
and store fatigue crack growth properties, and analyze fatigue crack
formation.
[34] When conducting structural assessments of the ISS, NASA assesses
each structural component with the same version of NASGRO.
[35] Actual loads are calculated by analytical modeling of the ISS
structure. The models are validated by data obtained from sensors on-
board the ISS.
[36] NASA engineers believe there is a reasonable expectation that
components that have analytically demonstrated more than 6 service
lives, or 90 years, are not likely to fail within the original service
life timeframe.
[37] NASA is conducting life extension structural and hardware
analyses through 2028 for two reasons: (1) the physical tests of the
FGB assessed its structural soundness through 2028 and, (2) the
program does not want to have to replicate the analysis if the ISS's
life is extended beyond 2020.
[38] These directives include NM 7120-97, NASA Procedural Requirement
7120.5D, SRB Handbook, NASA Space Flight Program and Project
Management Requirements, which is the interim directive to NPR
7120.5D; NASA Policy Directive 8700.1E, NASA Policy for Safety and
Mission Success, (October 28, 2008); NASA Policy Directive 8700.3B,
Safety and Mission Assurance Policy for NASA Spacecraft, Instruments,
and Launch Services, (October 28, 2008); Space Station Program 30599
Revision A, Safety Review Process (January 11, 1995); Space Station
Program 30309 Revision E, Safety Analysis and Risk Assessment
Requirements Document, (October 28, 1994).
[39] NASA's safety assessment covers only those ORUs whose failure
could result in loss of life or loss of station. These ORUs are a
subset of those covered by NASA's approach to determining spares
required to ensure utilization of the ISS through 2020 discussed
earlier in the report.
[40] Pub. L. No. 111-267.
[End of section]
GAO’s Mission:
The Government Accountability Office, the audit, evaluation, and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the
performance and accountability of the federal government for the
American people. GAO examines the use of public funds; evaluates
federal programs and policies; and provides analyses, recommendations,
and other assistance to help Congress make informed oversight, policy,
and funding decisions. GAO’s commitment to good government is
reflected in its core values of accountability, integrity, and
reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO’s website [hyperlink, http://www.gao.gov]. Each
weekday afternoon, GAO posts on its website newly released reports,
testimony, and correspondence. To have GAO e mail you a list of newly
posted products, go to [hyperlink, http://www.gao.gov] and select “E-
mail Updates.”
Order by Phone:
The price of each GAO publication reflects GAO’s actual cost of
production and distribution and depends on the number of pages in the
publication and whether the publication is printed in color or black
and white. Pricing and ordering information is posted on GAO’s
website, [hyperlink, http://www.gao.gov/ordering.htm].
Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.
Orders may be paid for using American Express, Discover Card,
MasterCard, Visa, check, or money order. Call for additional
information.
Connect with GAO:
Connect with GAO on facebook, flickr, twitter, and YouTube.
Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts.
Visit GAO on the web at [hyperlink, http://www.gao.gov].
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm];
E-mail: fraudnet@gao.gov;
Automated answering system: (800) 424-5454 or (202) 512-7470.
Congressional Relations:
Ralph Dawn, Managing Director, dawnr@gao.gov, (202) 512-4400
U.S. Government Accountability Office, 441 G Street NW, Room 7125
Washington, DC 20548.
Public Affairs:
Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, DC 20548.