This is the accessible text file for GAO report number GAO-07-745
entitled 'Defense Acquisitions: Success of Advanced SEAL Delivery
System Hinges on Establishing a Sound Contracting Strategy and
Performance Criteria' which was released on May 25, 2007.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Subcommittee on Emerging Threats and Capabilities,
Committee on Armed Services, U.S. Senate:
United States Government Accountability Office:
GAO:
May 2007:
Defense Acquisitions:
Success of Advanced SEAL Delivery System Hinges on Establishing a Sound
Contracting Strategy and Performance Criteria:
GAO-07-745:
GAO Highlights:
Highlights of GAO-07-745, a report to the Subcommittee on Emerging
Threats and Capabilities, Committee on Armed Services, U.S. Senate
Why GAO Did This Study:
The Advanced SEAL Delivery System (ASDS) is a hybrid combatant
submersible providing clandestine delivery and extraction of Navy SEALs
and equipment in high-threat environments. The first ASDS has had
significant performance issues and has cost, to date, over $885
million. In May 2006, you requested that GAO review ASDS. This report
examines (1) how the Navy managed ASDS risks through its contracts and
(2) the status of major technical issues and program restructuring.
What GAO Found:
The Navy did not effectively oversee the contracts to maintain, repair,
and upgrade the ASDS and failed to hold the prime contractor
accountable for results. The Navy took responsibility for correcting
the boat’s deficiencies while continuing to pay the costs and fees of
the prime contractor under cost reimbursable contracts to execute the
corrections. Before accepting the boat, the Navy went to sources other
than the prime contractor to obtain better designs for the propeller
and battery and then paid the prime contractor to install them. When
the Navy accepted the ASDS in 2003 in an “as is” condition, it relieved
the contractor from having to take any additional actions to correct
known problems. Since then, the U.S. Special Operations Command has
continued to invest millions of dollars to fix existing problems and
address new ones in an attempt to make the boat operational. In making
this additional investment, the Navy entered into contracts with the
prime contractor that provided little incentive to control costs,
authorized work before reaching agreement on the scope and price of the
work to be performed, and failed to finalize the terms of the work
within required time frames. Meanwhile, the contractor’s performance
continued to be poor, often exceeding initial estimates for the time
and cost required to perform the work. ASDS officials took actions over
the past 2 years to address these issues, but acknowledge that it is
too early to determine the effectiveness of more recent actions to
incentivize the contractor’s performance.
Continuing problems with the existing ASDS led to DOD’s April 2006
decision to cancel plans to buy additional ASDS boats, establish an
improvement program for the in-service ASDS, and conduct an assessment
of alternative material solutions to fulfill remaining operational
requirements. The problems have seriously degraded the boat’s
reliability and performance, and the boat is only available for limited
operational use. The results of these improvement and assessment
efforts are expected to provide DOD the knowledge needed to determine
whether ASDS’s reliability can be improved cost-effectively to make
ASDS an operational asset and whether an alternative development
program is needed to meet the remaining operational requirements. A
program decision is planned in mid-2008, after the ASDS improvement
program and assessment of alternate material solutions are completed.
What GAO Recommends:
GAO is making recommendations to the Secretary of Defense to help
ensure that a decision to proceed with ASDS is based on acceptable
cost, schedule, and performance criteria; that essential design changes
are operationally tested prior to a program decision; and that the
future contract strategy appropriately balances risk and promotes
better accountability. DOD partially concurred with GAO’s first two
recommendations and concurred with the third recommendation.
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-745].
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Paul Francis at (202) 512-
4841 or francisp@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
Navy Assumed Responsibility for ASDS Problems through Its Decisions and
Contracting Approach:
DOD Evaluating Future Options for Meeting Required Capabilities:
Conclusions:
Recommendations:
Agency Comments and Our Evaluation:
Scope and Methodology:
Appendix I: Mismatches in Technology, Resources, and Managerial
Capacity Undermined Key Business Case Assumptions:
Technology Assumptions Optimistic:
Initial Cost Projections Harbingers of Difficulties to Come:
Government and Contractor Management Was Ineffective:
Appendix II: Comments from the Department of Defense:
Figure:
Figure 1: ASDS Delivery Order Arrangements through March 2007:
Abbreviations:
ARAP: ASDS Reliability Action Panel:
ASDS: Advanced SEAL Delivery System:
BOA: basic ordering agreement:
DOD: Department of Defense:
FAR: Federal Acquisition Regulation:
JROC: Joint Requirements Oversight Council:
SOCOM: Special Operations Command:
United States Government Accountability Office:
Washington, DC 20548:
May 24, 2007:
The Honorable Jack Reed:
Chairman:
The Honorable Elizabeth Dole:
Ranking Member:
Subcommittee on Emerging Threats and Capabilities:
Committee on Armed Services:
United States Senate:
The Advanced SEAL Delivery System (ASDS), a hybrid combatant
submersible, is one of the U.S. Special Operations Command's (SOCOM)
largest investments. The ASDS is designed for clandestine delivery and
extraction of Navy SEALs and equipment in high-threat environments. In
2006, the Department of Defense's (DOD) Quadrennial Defense Review
Report revalidated the need for the type of capabilities that ASDS is
designed to provide.
The ASDS has encountered a difficult, long, and costly development
since the initial contract was awarded in 1994. The first ASDS boat did
not meet all technical or performance requirements, yet in 2003 it was
accepted by the Navy for operational use. Since acceptance, ASDS has
exhibited significant reliability and performance issues during test
and operation, and an in-service improvement effort has begun. For the
most part, the Navy has used the same contractor to design and deliver
the boat, to develop corrections to performance and reliability
problems, and to support the boat in the field. Total program costs--
including research and development, procurement, military construction,
operations and maintenance, and military personnel costs-
-have reached about $885 million.
In May 2006, you requested that GAO review the status and problems
facing the ASDS program. We subsequently agreed to address the issues
in two separate efforts. This report examines how the Navy managed ASDS
risks through its contracts and the status of major technical issues
and program restructuring.
Results in Brief:
The Navy accepted the ASDS in an "as is" condition to make the boat
operational, but it has not effectively overseen the contractor's
efforts to maintain, repair, and upgrade the boat, nor has it held the
contractor accountable for results, including the contractor's
inability to adhere to its own estimates of the time and cost to do the
work. As problems mounted during development and new problems arose
after acceptance, the Navy increasingly assumed responsibility for
their resolution. Further, the Navy's choice of contract type provided
little incentive to control costs; it authorized work before reaching
agreement on its scope and price; and it failed to finalize the terms
of the work when that was required. While ASDS officials reported they
have attempted over the past 2 years to address these issues, the
results of more recent efforts to incentivize the contractor's
performance are not yet clear.
Because of ASDS reliability and performance problems, the boat is only
available for limited operational use. These problems prompted DOD's
decision in April 2006 to cancel purchases of additional boats as well
as to establish a program to improve the in-service ASDS and assess
alternatives to fulfill remaining operational requirements. The results
of these two latter efforts are expected to help DOD determine if ASDS
can be made reliable enough to maintain it as an operational asset and
whether an alternative development program would be preferable. A
decision is planned in mid-2008.
We are making recommendations to the Secretary of Defense to help
ensure that a decision to proceed with ASDS is based upon acceptable
cost, schedule, and performance criteria; that essential design changes
are operationally tested prior to a program decision; and that the
future contract strategy appropriately balances risk and promotes
better accountability.
DOD partially concurred with our first two recommendations and
concurred with the third one. In partially concurring with our first
recommendation, DOD commented that under its new ASDS management plan,
program decisions will be made through management reviews using
specified evaluation criteria and not solely at the completion of the
critical systems reviews. We believe that, to be complete, such
criteria must be based on fully defined scopes of work, and we have
clarified the recommendation to include both the management reviews and
the stronger criteria. On the second recommendation, DOD noted that
some design changes may not be complete by the mid-2008 decision. We
have clarified our recommendation to include testing those design
changes essential to demonstrating ASDS reliability and
maintainability.
Background:
The special operations forces' ASDS is a battery-powered, dry interior
submersible that is carried to a deployment area by specially
configured 688-class submarines. ASDS is intended to provide increased
range, payload, on-station loiter time, endurance, and communication/
sensor capacity over current submersibles. The 65-foot-long, 8-foot-
diameter ASDS is operated by a two-person crew and includes a lock out/
lock in diving chamber.[Footnote 1] SOCOM is the resource sponsor and
provides the requirements and funding, and the Naval Sea Systems
Command--the Navy's technical expert for major undersea systems--is the
program manager responsible for overseeing the prime contractor,
Northrop Grumman Corporation. Over the years, the ASDS acquisition
milestone decision authority has resided at various levels within DOD.
In 1994, the Navy awarded a $70 million cost-plus incentive fee
contract[Footnote 2] to Westinghouse Electric Corporation's Oceanic
Division in Annapolis, Maryland, for detailed design, construction,
testing, documentation, refurbishment, and delivery of the first ASDS
with the option to build one or two more systems. In 1996, Northrop
Grumman bought this division and assumed responsibility for the
Annapolis division's performance on the ASDS contract. In December
2005, ASDS program management lead was reassigned to Northrop Grumman
in Newport News, Virginia, which has greater technical experience in
submarines, and Northrop Grumman Electronic Systems in Annapolis is
assisting.
The original program's schedule called for delivery of the first boat
in July 1997. However, numerous technical problems with key subsystems
contributed to performance shortfalls, schedule delays, and cost
increases. In August 2001, the Navy program office took what it called
"conditional" preliminary acceptance of the first boat from Northrop
Grumman under an agreement that all requirements needed for final
acceptance would be completed within 1 year, requirements that the
contractor was unable to accomplish. On June 26, 2003, the Navy elected
to accept the ASDS boat in an "as is" condition, and incorporated
additional waivers, deviations, and engineering change proposals into
the contract. As a result, acceptance of the ASDS boat did not require
any additional actions on the part of the contractor. Further, the Navy
did not seek any consideration from the contractor because Navy
officials believed at the time that the ASDS met virtually all of its
requirements. By that time, the total costs for the ASDS development
contract had already increased from $70 million to more than $340
million.
In October 2003, following the Navy's acceptance of ASDS, the Navy
negotiated and signed a basic ordering agreement (BOA)[Footnote 3] with
Northrop Grumman to provide a range of goods and services to support
the ASDS program. For example, the BOA enabled the Navy to order
engineering and design services; overhaul, repair, and inspection
services; logistical support; and spare parts and materials for a 3-
year period. The BOA was extended an additional year in 2006. To
expedite the contracting process, the BOA established specific labor
rates for different types of service, such as program office,
technical, engineering, operations, and quality support. Through March
2007, the Navy issued 26 delivery orders with an estimated value of
over $84 million. The duration of the current BOA extends through
September 2007, and the Navy anticipates awarding a new BOA for another
2 years while overall ASDS performance is reevaluated. Under another
BOA, Northrop Grumman is also providing ASDS engineering services, such
as engineering changes and drawing updates, for Portsmouth Naval
Shipyard.
In assessing the ASDS program we drew heavily from our previous work on
best practices in defense acquisitions. This work has shown that both a
sound business case and effective contracting strategy are essential
for success. A sound business case involves firm requirements and
mature technologies, a knowledge-based acquisition strategy, realistic
cost and schedule estimates, and sufficient funding. An effective
contracting strategy involves selecting a contractor with proper
expertise, choosing contracting approaches that effectively balance
risk, and effectively managing and assessing contractor performance;
all of which are intended to promote accountability for outcomes and
protect the taxpayers' interests.
Critical flaws in the Navy's initial business case contributed to
ASDS's acquisition challenges and increased the government's risk. We
have previously reported[Footnote 4] that the capabilities required of
the boat outstripped the contractor's resources in terms of technical
knowledge, time, and money. The Navy's overly optimistic assumptions
about the contractor's ability to readily incorporate existing
submersible and commercial technology into the ASDS resulted in a
mismatch between technologies and needed capabilities and an ill-
advised decision to combine developmental and operational testing.
Further information on the technical, cost, and management issues that
undermined the ASDS's initial business case may be found in appendix I.
Navy Assumed Responsibility for ASDS Problems through Its Decisions and
Contracting Approach:
As existing problems mounted during development and new ones arose
after acceptance, the Navy increasingly assumed responsibility for
resolving them. This responsibility required additional time and money
over the targets that had been established by the ASDS development
contract. Since accepting the ASDS in June 2003, SOCOM has continued to
invest millions of dollars to fix both old and new problems. The prime
contractor has had little incentive to control costs given the Navy's
choice of certain cost-reimbursable contract types. Navy officials say
they accept more risk of performance because ASDS relies on new, highly
technical subsystems that are inherently risky. The Navy's risk also
increased because it authorized work before reaching agreement on key
contract terms and conditions and failed to finalize them in a timely
manner, indicating a lack of discipline in the contracting process.
Over Time, the Navy Assumed the Cost and Responsibility for Correcting
ASDS Problems:
Resolving the flawed initial business case required additional time and
money, far exceeding the target cost and delivery time frames
established under the ASDS September 1994 development contract. For
example, the development contract was awarded for about $70 million
with an expected delivery date of the first ASDS boat in July 1997.
When the contractor proved unable to meet these time frames, the Navy
found itself having to rebaseline the program in 1998 and 1999, more
than doubling the estimated development cost and extending the delivery
schedule by more than 2 years. Ultimately the development cost almost
quintupled.
During the course of ASDS's development, the Navy gradually assumed
responsibility for addressing ASDS's technical problems by awarding
separate contracts to other organizations to develop key components.
The contractor's lack of expertise in key technologies, such as the
propeller and battery, contributed to the Navy's decision to seek
outside expertise to develop alternative solutions. More information on
these actions is provided in appendix I.
The Navy finally accepted the first ASDS boat in June 2003 in an "as
is" condition. Since the June 2003 acceptance, however, SOCOM has
continued to invest millions of dollars to address old and new
technical and reliability issues. Through March 2007, the Navy has
issued delivery orders with an estimated value of about $84 million
under the BOA with Northrop Grumman. Much of the funding has been for
efforts to correct design deficiencies and to improve ASDS's
reliability.
Navy's Cost-Reimbursable Contracts Provided Little Incentive to the
Contractor to Control Costs:
Arrangements that appropriately share risk, incentivize performance,
and provide for accountability promote successful acquisition outcomes.
The government can choose from a range of contract types available to
it that gives it flexibility to acquire goods and services. The
selection of contract type is generally a matter of risk allocation:
fixed-price contracts place the risks associated with performing the
contract on the contractor; cost-type contracts share the risk between
the contractor and the government. The risk associated with performance
shifts between the parties depending on the type of cost contract
selected. In selecting the contract type, the government must consider
the difficulty of providing the goods and services in the time
allocated for contract performance. For example, when the risks are
minimal or can be predicted with an acceptable degree of certainty,
such as when the government and the contractor have sufficient
knowledge of the effort required, then the government uses a fixed-
price contract, and the contractor has full responsibility for the
performance costs and the resulting profit or loss. In contrast, when
the extent of product knowledge is more limited, the government uses a
cost-reimbursable contract; the government assumes more risk and may
try to motivate the contractor's performance by using various incentive
or award fee provisions.
Our review found that nearly all of the $84 million in design,
integration, and reliability improvement work authorized under the
Navy's October 2003 BOA with Northrop Grumman used some form of a cost-
reimbursable contract. About 6 percent were conducted under a fixed-
price type arrangement. Of the first 18 delivery orders issued through
early May 2005, 14 were either cost-plus fixed fee or labor-hour
orders. Cost-plus fixed fee arrangements negotiate the fee at the
inception of the contract and do not vary with the actual costs
incurred by the contractor. Labor-hour contracts provide for direct
labor hours at specified fixed rates that include wages, overhead,
general, and administrative expenses. As profit and other expenses are
already included in the rates charged to the government, the orders
provided no profit incentive for the contractor to control costs or
work efficiently. Correspondingly, our analysis found that the ASDS
contractor often exceeded the initial estimates of the time and cost
required to complete the work: 12 of the 26 delivery orders issued
under the BOA exceeded the initial cost estimates, while the delivery
schedule was extended on 20 of the 26 orders. Figure 1 shows the value
of all delivery orders and subsequent modifications by contract type
through March 2007, based on the year the order was initially issued.
Figure 1: ASDS Delivery Order Arrangements through March 2007:
[See PDF for image]
Source: GAO analysis of ASDS delivery orders.
[End of figure]
Navy officials told us that they chose cost-plus fixed fee or labor-
hour orders, in part, because ASDS relied on many new and highly
technical subsystems that were inherently risky. The ASDS contracting
officer told us that the choice of cost-plus fixed fee or labor-hour
orders reflected the perceived risk in the efforts, that is, the
technical requirements and the work that needed to be done were not
always well-defined or known in advance. Navy officials reported,
however, that to get the contractor to more actively manage and be
accountable for success, the Navy has increased the use of award and
incentive fee provisions on its cost-type orders, placing at least some
of the contractor's potential fee at risk. For example, Navy officials
noted that two of the three delivery orders issued in 2006--
representing about 80 percent of the value of ASDS work ordered under
new delivery orders during the year--contained award or incentive fee
provisions. While the Navy officials acknowledged that it was too early
to quantify the results of these approaches, preliminary indications
are that the contractor's performance has improved and that the
arrangements are providing sufficient risk sharing and monetary
incentives to motivate contractor performance. Further, the contracting
officer anticipated that the Navy would use more fixed-price
arrangements as more experience is developed with ASDS repair and
maintenance requirements.
Authorizing Work before Reaching Agreement on Key Terms and Conditions
Increased the Navy's Risk:
Our analysis also found that the Navy often initiated work using
undefinitized contract actions; that is, before the Navy and contractor
had reached agreement on key terms and conditions of the delivery
order, such as the scope of the work to be performed and the price of
that work. While this approach allows agencies to begin needed work
quickly, it also exposes the government to potentially significant
additional costs and risk. For example, in September 2006 we reported
on how DOD addressed issues raised by the Defense Contract Audit Agency
in audits of Iraq-related contract costs.[Footnote 5] We found that DOD
contracting officials were less likely to remove costs questioned by
auditors if the contractor had already incurred those costs while the
contract action was undefinitized.
Our analysis found that 10 of the 26 ASDS delivery orders--accounting
for about 14 percent of the work--were initiated as undefinitized
contract actions. In most cases, the Navy justified the use of this
approach by stating that the work needed to begin immediately to meet
urgent operational requirements. For 7 of these 10 orders, the Navy
failed to definitize the orders within the 180-day time frame required
under defense acquisition regulations, taking instead from 228 to 509
days. In three cases, the Navy definitized the orders after the work
had been completed.
The delivery order to replace the ASDS's hydraulic reservoir
illustrates the need to clearly define the scope of the work, provide
effective management and oversight, and hold the contractor accountable
for outcomes. The delivery order issued to the contractor on June 10,
2005, was a $1.0 million cost-plus fixed fee undefinitized contract to
replace the ASDS's hydraulic reservoir. In October 2005, the contractor
reported it would need about $444,000 extra to complete the project.
Rather than provide additional funds, the Navy elected to reduce the
scope of the work, and the order was definitized on March 1, 2006--
nearly 9 months after the work was initially authorized--at a cost of
about $937,000. Two days later, the contractor reported that the
projected cost of the work had almost doubled to more than $1.85
million. In a letter to the contractor, the Commander, Naval Sea
Systems Command, noted that at no time during negotiations had the
contractor identified the potential cost growth. Nevertheless, as of
December 20, 2006, a further modification to the delivery order
increased the estimated cost to $2.8 million and extended the delivery
date by 60 days.
Navy officials acknowledged that the use of undefinitized contract
actions and the failure to definitize them in a timely fashion
indicated a lack of discipline in the contracting process, but noted
that officials had taken a number of actions to address the issues,
including taking more time to define requirements and requiring the
contractor to submit more realistic cost and schedule estimates.
Furthermore, the Navy has not issued an undefinitized contract action
since July 2005.
DOD Evaluating Future Options for Meeting Required Capabilities:
Continuing reliability problems led to a DOD decision to cancel
purchases of additional ASDS boats, following on an earlier decision to
decertify ASDS for operational test readiness because of considerable
performance and reliability issues that required significant additional
resources for new development, investigations, rework, and design
changes. Instead, DOD directed the establishment of an ASDS improvement
program and an assessment of alternate material solutions to fulfill
remaining operational requirements. The results of both should allow
DOD to make an informed decision as to its future needs by mid-2008.
Additional Procurements Canceled Because of Continuing Reliability
Problems:
The Navy decertified ASDS from operational test readiness in October
2005, following a propulsion-related failure during an attempt at
follow-on operational test and evaluation. This failure, however, was
among a series of performance and reliability issues identified over
the course of ASDS development. These performance and reliability
problems have required significant additional resources to support new
development, investigations, re-work, and design changes. Some changes
have not been fully corrected or verified in operational testing. For
example, in December 2003, while transporting ASDS mated to the host
submarine, severe damage occurred to the ASDS tail section--the
propeller assembly, the stator, and the stern planes. The Navy's
investigation attributed the cause to improper maintenance procedures-
-inadequate assembly by Portsmouth Naval Shipyard personnel. The
propeller assembly and stern plane designs were improved and
maintenance procedures were changed. In June 2004 testing of repairs,
however, the ASDS propeller stator broke off and damaged the propeller.
The investigation found that the stator had been improperly
manufactured by a subcontractor. The tail damage was repaired by
Northrop Grumman at the Navy's expense. During follow-on test and
evaluation in October 2005, ASDS experienced a propulsion system
failure that was attributed to improper assembly/installation of the
new titanium tail.
Because of the investigations of the December 2003 and June 2004 ASDS
tail casualties, the Navy re-evaluated the effects of unsteady
hydrodynamic loads on the boat. Although neither casualty was
attributed to this type of load, the Navy determined that, due to
fatigue stresses, the aluminum tail was not structurally adequate to
last the life of the ASDS. The tail was replaced with a titanium and
composite-based tail, but the replacement has not resolved all the tail
assembly design deficiencies. To minimize the potential for damage to
the tail, the Navy has imposed operating restrictions that limit the
speed of the host submarine while transporting ASDS, which will remain
in effect until this issue has been resolved.
In September 2005, the Navy and SOCOM chartered the ASDS Reliability
Action Panel (ARAP)--consisting of technical experts from government
and industry--to conduct an independent assessment of
reliability.[Footnote 6] After the 2005 propulsion system failure, the
ARAP was asked to assess ASDS's readiness to resume testing. ARAP's
report indicated that there were numerous examples of unpredicted
component reliability problems and failures resulting from design
issues, and recommended not resuming testing until detailed reviews of
mission critical systems were completed. In November 2005, SOCOM
restructured the ASDS program to focus on improving reliability of the
existing boat before investing in additional boats. The existing boat
is currently available only for limited operational use.
In April 2006, DOD canceled plans to procure follow-on ASDS boats and
directed the Navy and SOCOM to (1) establish an ASDS-1 improvement
program to increase the performance of the existing boat to the
required level, to insert technologies to avoid obsolescence, and to
complete operational testing and (2) assess alternate material
solutions to fulfill remaining operational requirements. In May 2006,
DOD reported to the congressional defense committees that the first
ASDS would be maintained as an operational asset, and that an ASDS
improvement program was planned through fiscal year 2008. As currently
structured, the ASDS reliability improvement program includes four
elements:
* ASDS Phase 1 and Phase 2 critical systems reviews,
* technical peer reviews,
* reliability builds or upgrades, and:
* verification testing.
The results of the Phase 1 critical systems review are due in June 2007
and are expected to include prioritized corrective actions and
associated cost and schedule estimates.[Footnote 7] According to Navy
officials, the Phase 1 results are expected to identify critical
upgrades to improve reliability and make ASDS-1 a viable operational
asset.[Footnote 8]
At-sea tests to verify that corrections result in improved performance
and reliability are being conducted. In October 2006 ASDS completed a
successful 2-week underway period operating from a host submarine to
verify and test repairs that were made to the propulsion system. In
February and March 2007, following installation of 15 reliability
improvements, including a newly designed hydraulic reservoir and
environmental control unit, ASDS verification testing was conducted.
This testing consisted of nine underways for a total 113 operating
hours. According to SOCOM, there were no failures. Follow-on
operational test and evaluation is scheduled for the second half of
fiscal year 2008. It is not certain, however, the extent to which the
upgrades identified by the Phase 1 critical systems review will be
incorporated into the ASDS for this operational test.
DOD Expects to Make a Program Decision in mid-2008:
DOD also directed the Navy and SOCOM to conduct an assessment of
alternate material solutions to fulfill remaining operational
requirements. An independent cost and capability trade study is under
way for the purpose of developing models for both the ASDS and a hybrid
combatant submersible to support concept design-level trade studies. A
final report is expected by the end of June 2007. SOCOM has completed a
requirements analysis that identified undersea clandestine maritime
mobility gaps for special operations forces insertion and extraction as
well as the conduct of undersea tasks. According to SOCOM, in February
2007, it submitted a memorandum on these issues to DOD's Joint Staff
for submission to the Joint Requirements Oversight Council (JROC). Upon
JROC approval, the memorandum is expected to serve in-lieu of an
Initial Capabilities Document for use in the alternate material
solutions analysis. This process is similar to an analysis of
alternatives and is expected to assess a broad range of potential
material solutions. The joint Navy-SOCOM alternate material solutions
analysis is expected to be completed by February 2008.
A program decision is planned in mid-2008, after the ASDS improvement
program and alternate material solutions analysis are completed.
According to SOCOM and Navy officials, the results of the alternate
material solutions analysis, in conjunction with the operational
testing of the changes made in response to the reliability improvement
program, should provide DOD by mid-2008 with sufficient information to
make an informed decision on the direction DOD should take to meet its
operational needs.
Conclusions:
Had the original business case for ASDS been properly assessed as an
under-resourced, concurrent technology, design, and construction effort
led by an inexperienced contractor, DOD may have adopted an alternative
solution or strategy. Ironically, after having invested about $885
million in nearly 13 years, DOD may still face this choice. As to
lessons learned, DOD's actions to make the boat operational came at
great expense to the government. Further, DOD's inadequate program and
contract management in essence made the prime contractor's poor
performance acceptable. These actions underscore the need to have a
sound business case at the start of a program, coupled with an
acquisition strategy that enables the government to alter course as
early as possible. Instilling more discipline into the contracting
process is a step in the right direction, but its success hinges on
DOD's willingness to hold the contractor accountable. From this point
forward, DOD will be conducting reviews and testing to guide its
decisions on how to proceed with the first ASDS boat. It is important
that DOD be guided by sound criteria and a sound contracting strategy
as it makes these decisions.
Recommendations:
We are making three recommendations. In order to prevent the government
from accepting additional undue risks and expense on ASDS, the
Secretary of Defense should:
* Establish acceptable cost, schedule, and performance criteria, based
on fully defined scopes of work, and assess the boat's ability to meet
these criteria at the Phase 1 and Phase 2 critical systems reviews and
at the management reviews. If, by the time of the program decision in
mid-2008, ASDS does not meet acceptable cost, schedule, or performance
criteria, we recommend that the Secretary of Defense discontinue the
effort and not proceed with further tests.
* Ensure that, if the review results meet acceptable cost, schedule,
and performance criteria, the design changes resulting from the Phase 1
critical systems review essential for demonstrating ASDS reliability
and maintainability be incorporated in sufficient time to be tested
under operational conditions prior to the planned mid-2008 decision on
how to best meet special operations forces' requirements.
* Require the Navy to include provisions in the ASDS contracting
strategy chosen when the existing BOA expires that (1) appropriately
balance risk between the government and the contractor through the
contract types selected, (2) incentivize the contractor's performance
and promote accountability for achieving desired outcomes by properly
structuring the award and incentive fees, and (3) provide the kind of
management and oversight of the program necessary to hold the
contractor accountable for performance.
Agency Comments and Our Evaluation:
DOD partially concurred with our first two recommendations that it
establish acceptable cost, schedule, and performance criteria for ASDS-
1; assess the boat's ability to meet these criteria; and test design
changes. DOD concurred with our third recommendation on the Navy's
contracting strategy to balance risk between the government and
contractor; properly structure award and incentive fees to incentivize
contractor performance and promote accountability; and provide
necessary management and oversight to hold the contractor accountable.
DOD's written comments are reprinted in appendix II.
In partially concurring with our first recommendation, DOD commented
that under its new ASDS management plan, program decisions will be made
through management reviews using specified evaluation criteria and not
solely at the completion of the critical systems reviews. The Navy
provided a copy of its March 6, 2007 management plan for ASDS-1
improvement. This plan represents a positive step in establishing a
structured strategy for the ASDS-1 improvement program, including
defining management oversight--roles, responsibilities, and
authorities--and providing specific criteria to guide the program's
continuation or termination decisions. However, the criteria may not go
far enough. Specifically, the criteria may not be sufficient for making
an informed program decision--the scope of the proposed ASDS's critical
systems upgrades may not be fully defined and realistic cost and
schedule estimates may not be developed before the ASDS improvement
effort is approved to proceed. Further, the management plan does not
address the Phase 2 critical systems review decision. We have clarified
this recommendation to incorporate the management program reviews and
decisions and added language to focus more directly on the need for
fully defined scopes of work. Fully defining the scopes of work is key
to realistic cost and schedule estimates.
DOD partially concurred with our second recommendation, but took issue
with operationally testing all Phase 1 critical systems review design
changes before the planned mid-2008 decision. DOD stated that there are
identified changes that will take more time and that a decision on what
changes to implement will depend on various factors such as time,
funding, and scope. However, it remains unclear the extent to which
upgrades that affect performance will be incorporated and tested prior
to the mid-2008 program decision. We modified the wording to require
testing essential design changes prior to a 2008 decision.
DOD also provided technical comments, which we have incorporated as
appropriate.
Scope and Methodology:
To assess the ASDS contracting strategy, we reviewed the ASDS
acquisition strategy, program documents, contract documentation, and
numerous historical documents, including the Navy's 1997 Independent
Review Team assessment, the joint Navy/SOCOM 1999 Independent Review
Team assessment, and the ASDS Reliability Action Panel's 2006 report.
In our assessment of ASDS, we drew upon our large body of previous work
on best practices for developing products and developing sound business
cases. To determine the status of major ASDS technical issues and
program restructuring, we examined program status documents and
briefings, test results, technical reports, and various memos and
guidance. We did not assess the appropriateness of accepting the first
ASDS boat in an "as is" condition. In performing our work, we obtained
information and interviewed officials from the U.S. Special Operations
Command; the Naval Sea Systems Command's ASDS program and contracting
offices; and the Navy's Operational Test and Evaluation Force. We
conducted our review from July 2006 to April 2007 in accordance with
generally accepted government auditing standards.
We are sending copies of this report to the Secretary of Defense; the
Secretary of the Navy; the Commander, U.S. Special Operations Command;
the Director of the Office of Management and Budget; and interested
congressional committees. We will make copies available to others upon
request. In addition, the report will be made available at no charge on
GAO's Web site at http://www.gao.gov.
If you or your staff have any questions about this report, please
contact me at (202) 512-4841 or by email at francisp@gao.gov. Contact
points for our Offices of Congressional Relations and Public Affairs
may be found on the last page of this report. Contributors to this
report include Catherine Baltzell, David Best, Timothy DiNapoli, David
Hand, John Krump, Mary Quinlan, and Robert Swierczek.
Signed by:
Paul L. Francis:
Director:
Acquisition and Sourcing Management:
[End of section]
Appendix I: Mismatches in Technology, Resources, and Managerial
Capacity Undermined Key Business Case Assumptions:
Putting a development program on sound footing from the beginning
requires that the selected technology be capable of meeting the
government's requirements and able to be developed within needed time
frames and available resources. Further, the contractor must have the
technical and managerial capacity to effectively execute the contract,
while the government must be able to provide effective program and
management oversight. On the ASDS program, however, these conditions
were not present at the start of or effectively applied during the
development effort, undermining the ability to successfully design and
deliver an operational ASDS boat.
Technology Assumptions Optimistic:
A key to promoting successful acquisition outcomes is matching
available resources with the requirements for the proposed system.
Specifically, the government must match its needs with technology that
has been proven to work in a realistic environment before committing to
production. In this case, the Navy assumed that the conceptual design
was technically sound and that the design would incorporate a large
amount of fully developed submersible or commercially available
technology. The Navy's September 1993 acquisition strategy concluded
that the low risk of integrating technologies already in use on
existing submarines and submersible vehicles eliminated the need for an
advanced development model or a demonstration/validation phase with
developmental and operational testing. Further, the Navy determined
that by concurrently addressing manufacturing and test issues during
the design process, lengthy redesign periods would be avoided.
Consequently, in September 1994, the Assistant Secretary of the Navy
for Research, Development, and Acquisition (the designated program
decision authority) approved Milestone II (development) and replaced a
sequential test program (development tests, operational tests,
technical evaluations, and operational test and evaluation) with a
consolidated and integrated test program. At the same time, the ASDS
program's Milestone III (production decision) was waived because of the
limited number of procurement quantities.
The Navy's confidence in the maturity of technology also played a large
role in its assessment of proposed designs for the ASDS, and in turn,
in its selection of the contractor. The Navy concluded that the
contractor's conceptual design exceeded various requirements, and,
based on its maturity, the proposed design approach was low
risk.[Footnote 9] From the outset, the Navy's assessments of the
contractor's design solution, experience, and management capabilities
proved incorrect. Incorporating commercial off the shelf components
into the ASDS was more challenging than expected. For example, the
contractor had difficulty understanding underwater shock performance
requirements and eventually subcontracted the shock design efforts to a
specialty firm.
During the course of ASDS's development, the Navy gradually assumed
responsibility for addressing ASDS's technical problems by awarding
separate contracts to other organizations to develop key components.
The contractor's lack of expertise in key technologies, such as the
propeller and battery, contributed to the Navy's decision to seek
outside expertise to develop alternative solutions. In turn, the Navy
provided these components to Northrop Grumman as government-furnished
equipment, accepting both the cost and the risk for their performance
and paid Northrop Grumman millions of dollars to integrate the
components onto the ASDS boat. These actions include the following
examples:
* The ASDS program has invested over $26 million since 2000 to design,
develop, and integrate a new lithium-ion battery to replace the
inadequate silver-zinc battery provided by the prime contractor. In
October 2000, the Navy awarded Northrop Grumman a $2.1 million contract
modification to design, develop, test, and incorporate a lithium-ion
polymer battery. By September 2003, a series of contract modifications
had increased the cost of the prototype battery effort to $5.9 million
and had extended delivery until February 28, 2004. The Navy sought
other experts to identify and test an alternative lithium-ion battery
that could be housed in the existing ASDS titanium battery bottles. In
May 2004, after evaluating three proposals, the Navy awarded Yardney
Technical Products a $9.3 million contract for a complete ASDS shipset
battery that was delivered in 2005. To date, the Navy has provided
Northrop Grumman more than $6 million to integrate the lithium ion
battery.
* The Navy invested over $1.5 million to redesign the first ASDS
propeller, which was a major source of noise during testing. Rather
than task Northrop Grumman to redesign the propeller, the Navy awarded
a $1.5 million contract in 2002 to Pennsylvania State University's
Applied Research Laboratory to design and build a new composite
propeller. Northrop Grumman installed this propeller in April 2003 at a
cost of about $140,000. Pennsylvania State University has since
provided two additional propellers at a cost of about $576,000.
Initial Cost Projections Harbingers of Difficulties to Come:
Another key to successful acquisition outcomes is to accurately
estimate the resources needed to develop and produce a system. The Navy
had information before awarding the ASDS contract indicating, however,
that the contractor's proposed price might not be realistic.
Specifically, the contract's negotiated price was about 60 percent less
than the Navy's November 1993 cost and operational effectiveness
analyses.[Footnote 10] The Navy's price evaluation team concluded that
the contractor's proposed amounts for ASDS development and production
were underestimated and that overruns were likely. Among the lessons
learned cited by two independent review teams in 1997 and 1999 were
that the program was underfunded, in part because the Navy did not give
sufficient weight to concerns raised by cost analysts, and that the
contractor "bid to the budget."
Government and Contractor Management Was Ineffective:
The government's and contractor's capacity to effectively manage a
program is another key determinant in promoting successful outcomes.
The Navy concluded in 1994 that overall, the contractor's design,
management capabilities, and cost control capabilities were equal to or
better than the two other competitors for the ASDS program and that the
contractor had adequate experience in submersible design, construction,
and certification. This assessment, as well as the government's
capacity to provide effective management and oversight of the ASDS
program, soon proved incorrect.
The Navy's 1997 and the joint Navy/SOCOM 1999 independent review teams
identified weaknesses in the contractor's capacity to effectively
address technical issues and manage the ASDS program. One team noted
that the contractor had considerable difficulty in interpreting the
underwater shock portion of the ASDS performance requirements. The
teams attributed these difficulties, in part, to the contractor's lack
of experience in submarine design, in contrast to the initial business
case assumption. Further, the reviews noted that the Navy's review of
the contractor's design products revealed that substandard design
methodology was used, resulting in unacceptable system design. The
review teams also found that this lack of experience had a detrimental
effect on the contractor's overall ability to understand technical
nuances and may have prevented the contractor from applying appropriate
management attention when needed. For example, the contractor used two
different systems for reporting and managing the program; the
contractor's cost reports contained errors; and its estimates to
complete the effort were updated only every 6 months, resulting in
unanticipated and sudden cost increases being reported to the Navy.
Additionally, the contractor constrained its estimates by imposing
"management challenges," which the team concluded were in reality
artificial reductions imposed by the contractor to obscure the
contractor's problems and mislead attempts to analyze its projected
costs.
Further, the review teams concluded that lapses in effective management
by both the government and the contractor contributed to the program's
challenges. The teams identified several causes for these lapses,
including:
* a lack of contractor experience in submarine design and construction;
* the government's lack of influence or visibility into problems
between the contractor and the subcontractors;
* a focus on technical rather than management aspects of the program by
both the program office and the contractor;
* ineffective oversight by the program office and little attention to
the financial performance of the contractor; and:
* frequent changes in the contractor's project management team.
The Navy program office and Northrop Grumman have taken steps to
improve the program's management. In 2005, the Naval Sea Systems
Command reorganized the program office for a greater emphasis on
special operations programs. In December 2005, Northrop Grumman
reassigned the ASDS program's management lead to its Newport News
division, which has greater management and technical experience in
submarines. Northrop Grumman Newport News is leading the Phase 1 ASDS
critical systems review, and Northrop Grumman Electronic Systems is
assisting.
[End of section]
Appendix II: Comments from the Department of Defense:
Office Of The Under Secretary Of Defense:
3000 Defense Pentagon:
Washington, DC 20301-5000:
Acquisition, Technology And Logistics:
May 17 2002:
Paul L. Francis:
Director, Acquisition and Sourcing Management:
U.S. General Accountability Office:
441 G Street, N.W.
Washington, D.C. 20548:
Dear Mr. Francis:
This is the Department of Defense (DoD) response to the GAO Draft
Report, GAO-07-745, `Defense Acquisitions: Success of Advanced SEAL
Delivery System Hinges on Establishing a Sound Contracting Strategy and
Performance Criteria,' dated April 27, 2007 (GAO Code 120525).
The Department partially concurs with the statements in the draft
report, partially concurs with recommendations one and two, and concurs
with recommendation three.
The detailed response to the report recommendations are provided in the
enclosure. Suggested technical changes have been provided separately.
Thank you for the opportunity to comment on the draft report.
Sincerely,
Signed by:
David G. Ahern:
Director:
Portfolio Systems Acquisition:
Enclosure:
As stated:
GAO Draft Report - Dated April 27, 2007 GAO-07-745 (GAO Code 120525):
"Defense Acquisitions: Success Of Advanced Seal Delivery System Hinges
On Establishing A Sound Contracting Strategy And Performance Criteria"
Department Of Defense Comments To The Recommendations:
Recommendation 1: The GAO recommended that the Secretary of Defense
establish acceptable cost, schedule and performance criteria for the
first advanced SEAL delivery system (ASDS) boat and assess the boat's
ability to meet these values at the phase 1 and phase 2 critical
systems reviews. If at either of these reviews ASDS does not meet
acceptable cost, schedule or performance criteria, the GAO recommended
that the Secretary of Defense discontinue the effort and not proceed
with the 2008 test. (p. 11 /GAO Draft Report):
DoD Response: The DoD partially concurs with this recommendation. The
Department agrees that there needs to be acceptable cost, schedule and
performance criteria. The Navy and Commander, USSOCOM have developed a
management plan which adequately details evaluation criteria and
establishes routine management program reviews with off ramps. It is at
these reviews, vice solely at the completion of the Critical System
Reviews (CSR), that future program status will be determined.
Recommendation 2: The GAO recommended that the Secretary of Defense
ensure that, if it does meet acceptable cost, schedule, and performance
criteria, that any design changes resulting from the Phase 1 critical
systems review be incorporated in sufficient time to be tested under
operational conditions prior to the planned mid-2008 decision as to how
to best meet special operational forces requirements. (p. 11/GAO Draft
Report):
DoD Response: The DoD partially concurs with this recommendation. The
Department takes issue with the use of the word "any" in the first
sentence. There are identified design changes (e.g. hydro-dynamic
mitigation) that will not be complete by the mid-2008 decision due to
the time it takes to develop, build, install, and test. The decision on
which CSR design changes are implemented and installed will depend upon
the relative improvement expected, scope of the effort needed, as well
as available time and funding. Testing might be accomplished without
some recommended improvements being made to gauge progress and
acceptability of the vehicle, and to support an overall program
decision. Additionally, unforeseen program issues could result in a
decision to discontinue the ASDS effort without any additional
operational testing.
Recommendation 3: The GAO recommended that the Secretary of Defense
require the Navy to include provisions in the ASDS contracting strategy
chosen when the existing basic order agreement expires that (1)
appropriately balance risk between the government and the contractor
through the contract types selected, (2) incentivize the contractor's
performance and promotes accountability for achieving desired outcomes
by properly structuring the award and incentive fees, and (3) provide
the kind of management and oversight of the program necessary to hold
the contractor accountable for performance. (p. 11/GAO Draft Report):
DoD Response: The DoD concurs with this recommendation. The Department
intends to continue the maximized use of incentivized contract efforts
to conduct Phase 2 CSR and implementation of reliability improvements
into ASDS-l. The incentivized performance will balance risk and promote
accountability. Senior department personnel will continue to manage and
oversee the program through these phases.
[End of section]
FOOTNOTES
[1] A chamber used to treat divers suffering from decompression
sickness, which can be caused by descending below sea level.
[2] A cost-plus incentive fee contract as a cost-reimbursement contract
that provides for the amount a contractor earns as profit or fee to be
adjusted based on the contractor's ability to meet established cost
targets. Federal Acquisition Regulation (FAR) 16.304.
[3] A basic ordering agreement (BOA) is a written understanding between
a government entity and a contractor that contains the terms and
clauses applying to future orders; describes the types of supplies and
services to be provided; and contains the methods for providing,
issuing, and delivering future orders. A BOA is not a contract. FAR
16.703. For example, a BOA differs from a standard contract in that the
specific goods and services to be provided, their cost, and the
delivery schedule or period of performance are not established until an
order is issued.
[4] GAO, Defense Acquisitions: Advanced SEAL Delivery System Program
Needs Increased Oversight, GAO-03-442 (Washington, D.C.: Mar. 31,
2003).
[5] GAO, Iraq Contract Costs: DOD Consideration of Defense Contract
Audit Agency's Findings, GAO-06-1132 (Washington, D.C.: Sept. 25,
2006).
[6] The charter for the ARAP is to provide an independent assessment of
ASDS reliability and provide recommended ASDS design, process, and
procedural changes to improve reliability or mitigate shortfalls.
[7] The Phase 1 program is intended to address seven systems, including
(1) hydraulics; (2) battery and power distribution; (3) main propulsion
and thruster maneuvering; (4) life support; (5) variable and freeboard
ballast; (6) sensors; and (7) host submarine interface. The Phase 1
program will also address five cross-system technical areas:
hydrodynamics (based on a study of the entire vehicle, across all host
platforms); vibration; naval architecture (weights); electromagnetic
compatibility; and corrosion issues.
[8] The Phase 2 critical systems review is in the planning stage but
expected to be complete in December 2007. Technical Peer reviews are
ongoing throughout the improvement program. One ASDS Reliability build
or upgrade has been completed and two more are planned through 2008.
[9] In contrast, while noting that the other two competitors had
significant experience in designing, building, and testing submarines,
the Navy identified weaknesses in their proposed designs that would
have required significant government assistance.
[10] It should be noted that the other two competing contractors'
proposals were also within the same cost range-well below the
government's estimate.
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site (www.gao.gov). Each weekday, GAO posts
newly released reports, testimony, and correspondence on its Web site.
To have GAO e-mail you a list of newly posted products every afternoon,
go to www.gao.gov and select "Subscribe to Updates."
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm:
E-mail: fraudnet@gao.gov:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400:
U.S. Government Accountability Office, 441 G Street NW, Room 7125
Washington, D.C. 20548:
Public Affairs:
Paul Anderson, Managing Director, AndersonP1@gao.gov (202) 512-4800:
U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548: