Additional Materials:

Contact:

GAO has designated the Department of Defense's (DOD) business systems modernization as a high-risk program because, among other things, it has been challenged in implementing key information technology (IT) management controls on its thousands of business systems. The Global Combat Support System-Marine Corps program is one such system. Initiated in 2003, the program is to modernize the Marine Corps logistics systems. The first increment is to cost about $442 million and be deployed in fiscal year 2010. GAO was asked to determine whether the Department of the Navy is effectively implementing IT management controls on this program. To accomplish this, GAO analyzed the program's implementation of several key IT management disciplines, including economic justification, earned value management, risk management, and system quality measurement.

DOD has not effectively implemented key IT management controls provided for in DOD and related acquisition guidance on this program. If implemented effectively, these and other IT management disciplines increase the likelihood that a given system investment will produce the right solution to fill a mission need and that this system solution will be acquired and deployed in a manner that maximizes the chances of delivering promised system capabilities and benefits on time and within budget. Neither of these outcomes is being fully realized on this program, as evidenced by the fact that its first increment has already slipped more than 3 years and is expected to cost about $193 million more than envisioned. These slippages and cost overruns can be attributed in part to the management control weaknesses discussed in this report and summarized below. Moreover, additional slippages and overruns are likely if these and other IT management weaknesses are not addressed. Investment in the system has not been economically justified on the basis of reliable estimates of both benefits and costs. Specifically, while projected benefits were risk-adjusted to compensate for limited data and questionable assumptions, the cost side of the benefit/cost equation is not sufficiently reliable because it was not derived in accordance with key cost estimating practices. In particular, it was not based on historical data from similar programs and it did not account for schedule risks, both of which are needed for the estimate to be considered accurate and credible. Earned value management that the program uses to measure progress has not been adequately implemented. Specifically, the schedule baseline against which the program gauges progress is not based on key estimating practices provided for in federal guidance, such as assessing schedule risks and allocating schedule reserves to address these risks. As a result, program progress cannot be adequately measured, and likely program completion dates cannot be projected based on actual work performed. Some significant program risks have not been adequately managed. While a well-defined risk management plan and supporting process have been put in place, the process has not always been followed. Specifically, mitigation steps for significant risks either have not been implemented or proved ineffective, allowing the risks to become actual problems. The data needed to produce key indicators of system quality, such as trends in the volume of significant and unresolved problems and change requests, are not being collected. Without such data, it is unclear whether the system is becoming more or less mature and stable. The reasons for these weaknesses range from limitations of DOD guidance and tools, to not collecting relevant data. Until they are addressed, DOD is at risk of delivering a solution that does not cost-effectively support mission operations and falls short of cost, schedule, and capability expectations.

Recommendations for Executive Action

Status: Closed - Implemented

Comments: DOD has taken a number of actions to strengthen risk management, in addition to revising its Risk Management Plan in March 2009 to reflect new processes and policies. First, DOD reported that the program office inserted all risks reported by GAO, as well as risks identified through DOD's Enterprise Risk Assessment Methodology, into its risk database. Second, the Risk Board, chaired by the program manager and including senior program leadership, reviews on a monthly basis, new program risks, the status of existing high risks, and management of risk mitigation activities. Third, the program meets monthly with the Program Executive Office for Enterprise Information Systems, and quarterly with the Assistant Secretary of the Navy (Research Development and Acquisition), to discuss program risks. Also, the program reports risks to other program oversight bodies, such as the Weapons Systems Lifecycle Management and Materiel Supply and Services Management Investment Review Board, and the Defense Business Systems Management Committee. Finally, risks are not closed without risk board approval.

Recommendation: To improve GCSS-MC management of program risks, the Secretary of Defense should direct the Secretary of the Navy, through the appropriate chain of command, to ensure that the program office (1) adds each of the risks discussed in this report to its active inventory of risks, (2) tracks and evaluates the implementation of mitigation plans for all risks, (3) discloses to appropriate program oversight and approval authorities whether mitigation plans have been fully executed and have produced the intended outcome(s), and (4) only closes a risk if its mitigation plan has been fully executed and produced the intended outcome(s).

Agency Affected: Department of Defense

Status: Closed - Not Implemented

Comments: As of September 2012, the Department of Defense has made progress addressing our recommendation, but work remains. For example, the GCSS-MC program office reported that it has begun to monitor and report planned versus actual start and complete dates of work activities, and assigns resources to the schedule to project the level of effort needed to accomplish detailed program activities. In addition, the department has disclosed the risk associated with its schedule in using EVM reports, in accordance with our recommendation. However, key activities in program sub-schedules have yet to be integrated, which has led to schedule anomalies and inconsistency between the sub-schedules, and an inability to identify a critical path. Further, while the schedule includes risk-related activities, the program has yet to conduct a quantitative schedule risk analysis or allocate schedule risk reserve for high-risk activities. By not performing these practices the schedule does not provide a reliable basis for performing EVM.

Recommendation: To enhance GCSS-MC's use of EVM, the Secretary of Defense should direct the Secretary of the Navy, through the appropriate chain of command, to ensure that the program office (1) monitors the actual start and completion dates of work activities performed so that the impact of deviations on downstream scheduled work can be proactively addressed; (2) allocates resources, such as labor hours and material, to all key activities on the schedule; (3) integrates key activities and supporting tasks and subtasks; (4) identifies and allocates the amount of float time needed for key activities to account for potential problems that might occur along or near the schedule's critical path; (5) performs a schedule risk analysis to determine the level of confidence in meeting the program's activities and completion date; (6) allocates schedule reserve for high-risk activities on the critical path; and (7) discloses the inherent risks and limitations associated with any future use of the program's EVM reports until the schedule has been risk-adjusted.

Agency Affected: Department of Defense

Status: Closed - Implemented

Comments: The department took steps to improve the credibility of its cost estimate in accordance with our recommendation. For example, while it has yet to use the cost estimate for its economic analysis, the department has applied risk and uncertainty to hardware refresh and software license costs based on the variability in prices experienced in other programs. According to officials, an updated economic analysis is planned to be completed for the program's Full Deployment Decision in December 2012.

Recommendation: To improve the credibility of the GCSS-MC cost estimate, the Secretary of Defense should direct the Secretary of the Navy, through the appropriate chain of command, to ensure that the program's current economic analysis is adjusted to reflect the risks associated with it not reflecting cost data for comparable ERP programs, and otherwise not having been derived according to other key cost estimating practices, and that future updates to the GCSS-MC economic analysis similarly do so.

Agency Affected: Department of Defense

Status: Closed - Implemented

Comments: DOD has largely implemented the recommendation, as intended. Specifically, the department established in October 2011 a standard practice for creating work breakdown structures, which includes common elements for ERP programs such as GCSS-MC, and reported that it plans to continue to establish common practices for ERP programs in the area of cost. Further, DOD reported that it expects the benefits of increased accuracy in cost estimating to be realized in the future when the standard WBS practice is implemented cross the department and additional data is collected in the cost elements.

Recommendation: To improve the accuracy of the GCSS-MC cost estimate, as well as other cost estimates for the department's enterprise resource planning (ERP) programs, the Secretary of Defense should direct the appropriate organization within DOD to collaborate with relevant organizations to standardize the cost element structure for the department's ERP programs and to use this standard structure to maintain cost data for its ERP programs, including GCSS-MC, and to use this cost data in developing future cost estimates.

Agency Affected: Department of Defense

Status: Closed - Implemented

Comments: DOD has addressed the intent of our recommendation. Specifically, the department conducted an Enterprise Risk Assessment Methodology (ERAM)-based review in February 2009, to support the May 2010 Milestone C decision to begin the program's Production and Deployment phase, and reported in March 2011 that all of the risks except one identified in the review are closed. Since then, the department reported that it closed the remaining issue, related to having adequate staff to support multiple software releases. The ERAM process identified seven risk areas, some of which relate to risks discussed in our report, such as not having a well-developed schedule. The assessment included a review of the program's risk management database and policies, and the results were presented at a May 2009 investment review board meeting. In addition, DOD developed metrics related to system maturity and stability, such as metrics to track help desk tickets and change requests, and is generating monthly trend analyses of each. Finally, the department has made progress in addressing weaknesses in its management control practices, including cost estimating.

Recommendation: To ensure that each Global Combat Support System-Marine Corps (GCSS-MC) system increment is economically justified on the basis of a full and reliable understanding of costs, benefits, and risks, the Secretary of Defense should direct the Secretary of the Navy to ensure that investment in the next acquisition phase of the program's first increment is conditional upon fully disclosing to program oversight and approval entities the steps under way or planned to address each of the risks discussed in this report, including the risk of not being architecturally compliant and being duplicative of related programs, not producing expected mission benefits commensurate with reliably estimated costs, not effectively implementing earned value management (EVM), not mitigating known program risks, and not knowing whether the system is becoming more or less mature and stable. We further recommend that investment in all future GCSS-MC increments be limited if the management control weaknesses that are the source of these risks, and which are discussed in this report, have not been fully addressed.

Agency Affected: Department of Defense

Status: Closed - Implemented

Comments: This recommendation has largely been implemented as intended. The program office is now collecting data according to their priority and severity and developing trends over time in open and closed help desk tickets and change requests. These trends are reported to the III Marine Expeditionary Force; Director, Operational Test and Evaluation; Program Executive Officer--Enterprise Information Systems and the Marine Corps Installation and Logistics Director.

Recommendation: To strengthen GCSS-MC system quality measurement, the Secretary of Defense should direct the Secretary of the Navy, through the appropriate chain of command, to ensure that the program office (1) collects the data needed to develop trends in unresolved system defects and change requests according to their priority and severity and (2) discloses these trends to appropriate program oversight and approval authorities.