Additional Materials:

Contact:

The Decennial Census is mandated by the U.S. Constitution and provides vital data that are used, among other things, to reapportion and redistrict congressional seats. In March 2008, GAO designated the 2010 Decennial Census a high-risk area, citing a number of long-standing and emerging challenges, including weaknesses in the Census Bureau's (Bureau) management of its information technology (IT) systems and operations. In conducting the 2010 census, the Bureau is relying on both the acquisition of new IT systems and the enhancement of existing systems. Thoroughly testing these systems before their actual use is critical to the success of the census. GAO was asked to determine the status of and plans for testing key decennial systems. To do this, GAO analyzed testing documentation, interviewed Bureau officials and contractors, and compared the Bureau's efforts with recognized best practices.

Although the Bureau has made progress in testing key decennial systems, critical testing activities remain to be performed before systems will be ready to support the 2010 census. Bureau program offices have completed some testing of individual systems, but significant work still remains to be done, and many plans have not yet been developed (see table below). In its testing of system integration, the Bureau has not completed critical activities; it also lacks a master list of interfaces between systems; has not set priorities for the testing of interfaces based on criticality; and has not developed testing plans and schedules. Although the Bureau had originally planned what it refers to as a Dress Rehearsal, starting in 2006, to serve as a comprehensive end-to-end test of key operations and systems, significant problems were identified during testing. As a result, several key operations were removed from the Dress Rehearsal and did not undergo end-to-end testing. The Bureau has neither developed testing plans for these key operations, nor has it determined when such plans will be completed. Weaknesses in the Bureau's testing progress and plans can be attributed, in part, to a lack of sufficient executive-level oversight and guidance. Bureau management does provide oversight of system testing activities, but the oversight activities are not sufficient. For example, Bureau reports do not provide comprehensive status information on progress in testing key systems and interfaces, and assessments of the overall status of testing for key operations are not based on quantitative metrics. Specifically, key operations that do not yet have plans developed are marked as making acceptable progress based solely on management judgment. Further, although the Bureau has issued general testing guidance, it is neither mandatory nor specific enough to ensure consistency in conducting system testing. Without adequate oversight and more comprehensive guidance, the Bureau cannot ensure that it is thoroughly testing its systems and properly prioritizing testing activities before the 2010 Decennial Census, posing the risk that these systems may not perform as planned.

Recommendations for Executive Action

Status: Closed - Implemented

Comments: The Census Bureau finalized the system requirements for UC&M and RPS and established test plans that trace to the system requirements.

Recommendation: To ensure that testing activities for key systems for the 2010 census are completed, the Secretary of Commerce should require the Director of the Census Bureau to expeditiously, for the Headquarters Universe Control & Management (UC&M) and Response Processing System (RPS), finalize requirements for 2010 census operations and complete testing plans and schedules for 2010 operations that trace to baselined system requirements.

Agency Affected: Department of Commerce

Status: Closed - Implemented

Comments: The Census Bureau has determined the number of MAF/TIGER products required for 2010 operations and baselined related test plans and schedules. In addition, the Bureau has conducted Independent Testing and Verification to better ensure that the MAF/TIGER system software meets both its functional and system requirements.

Recommendation: To ensure that testing activities for key systems for the 2010 census are completed, the Secretary of Commerce should require the Director of the Census Bureau to expeditiously, for Master Address File/Topologically Integrated Geographic Encoding and Referencing system (MAF/TIGER), establish the number of products required, define related requirements, and establish a testing plan and schedule for 2010 operations.

Agency Affected: Department of Commerce

Status: Closed - Implemented

Comments: The Bureau has established test plans for continuity of operations and map printing systems.

Recommendation: To ensure that testing activities for key systems for the 2010 census are completed, the Secretary of Commerce should require the Director of the Census Bureau to expeditiously, for Field Data Collection Automation (FDCA), establish testing plans for the continuity of operations and map printing systems that trace to baselined system requirements.

Agency Affected: Department of Commerce

Status: Closed - Implemented

Comments: The bureau has developed high-level requirements and established schedules for the system. In addition, the Bureau has developed a test plan which documents the testing strategy that will be used to verify and ensure that the system meets the developed requirements.

Recommendation: To ensure that testing activities for key systems for the 2010 census are completed, the Secretary of Commerce should require the Director of the Census Bureau to expeditiously, for Paper-Based Operations (PBO), develop baseline requirements and complete testing plans and schedules for 2010 operations.

Agency Affected: Department of Commerce

Status: Closed - Implemented

Comments: The Bureau has updated their master list of interfaces to define all interfaces. In addition, the Bureau has developed a process to track the progress of testing all interfaces.

Recommendation: To ensure that testing activities for key systems for the 2010 census are completed, the Secretary of Commerce should require the Director of the Census Bureau to expeditiously establish a master list of system interfaces; prioritize the list, based on system criticality and need date; define all interfaces; and develop integration testing plans and schedules for tracking the progress of testing these interfaces.

Agency Affected: Department of Commerce

Status: Closed - Implemented

Comments: The Census Bureau finalized test plans and completed testing for most operations removed from the Dress Rehearsal. These operations include Group Quarters Enumeration, Nonresponse Follow-up, Enumeration at Transitory Locations, and Census Coverage Measurement.

Recommendation: To ensure that testing activities for key systems for the 2010 census are completed, the Secretary of Commerce should require the Director of the Census Bureau to expeditiously establish a date for completing testing plans for the operations removed from the Dress Rehearsal operations and prioritize testing activities for these operations.

Agency Affected: Department of Commerce

Status: Closed - Implemented

Comments: The Bureau finalized its plan for operational and system failures.

Recommendation: To ensure that testing activities for key systems for the 2010 census are completed, the Secretary of Commerce should require the Director of the Census Bureau to expeditiously finalize risk mitigation plans detailing actions to address system problems that are identified during testing.

Agency Affected: Department of Commerce

Status: Closed - Implemented

Comments: The bureau has required specific metrics and detailed reports to be presented to management for all operations and systems. In addition, the bureau has developed a testing overview which provides detailed status of testing for all systems.

Recommendation: To ensure that testing activities for key systems for the 2010 census are completed, the Secretary of Commerce should require the Director of the Census Bureau to expeditiously establish specific testing metrics and detailed status reports to monitor testing progress and better determine whether corrective action is needed for all key testing activities.

Agency Affected: Department of Commerce

Status: Closed - Implemented

Comments: The Bureau developed a document titled "2020 Census Research and Testing (R&T) Phase Test and Evaluation Management Plan (TEMP) dated May 21, 2013 that provides guidance to those groups that are supporting the testing and evaluation of 2020 systems or have a role in enabling them. The TEMP outlines that 2020 R&T system development projects must have a documented Master Test Plan. The plan requirements include a description of the levels of testing to be planned and executed to verify and validate the solution. The plan must include the conditions for determining when testing or evaluation is successfully completed at all levels of testing. Further, the TEMP outlines requirements for preparing test and evaluation products, such as the Integration Test Plan. The Bureau states that the TEMP will evolve as it moves into the System Development Life Cycle phases of the 2020 lifecycle.

Recommendation: To ensure that testing activities for key systems for the 2010 census are completed, the Secretary of Commerce should require the Director of the Census Bureau to expeditiously designate a senior manager with primary responsibility for monitoring testing and overseeing testing across the Bureau.

Agency Affected: Department of Commerce

Status: Closed - Implemented

Comments: As of September 2013, the Bureau issued a Test Evaluation Management Plan (TEMP) to guide systems testing during the 2020 Research and Test Phase. Census has also defined a test metrics process that will begin in Q1 2014 and iteratively improved via lessons learned. According to the Bureau, an update to the SDLC guidelines, called SCLC 2.0, will be baselined in October 2013 and required for all new IT projects at that point in time. After Enterprise SDLC 2.0 is issued, the Bureau will be able to update its TEMP to follow those testing guidelines, including I & T testing guidelines. The Bureau plans for those updates to be available in March 2014.

Recommendation: After the 2010 census, the Bureau should establish comprehensive systems and integration testing guidance to guide future testing of systems.