Additional Materials:

Contact:

In 2003, the Centers for Disease Control and Prevention (CDC), an agency within the Department of Health and Human Services (HHS), developed an electronic syndromic surveillance system called BioSense that uses health-related data to identify patterns of disease symptoms prior to specific diagnoses. In late 2007, CDC began to redesign the program to improve collaboration with stakeholders and address identified management weaknesses. Pursuant to House Report 110-231, GAO evaluated the BioSense program, focusing on the cost and timeline estimates and performance measures and benchmarks for implementing the program, among other objectives. To accomplish this, GAO analyzed relevant program documentation and interviewed CDC officials responsible for developing and implementing BioSense.

While CDC identified annual and long-term cost and timeline estimates and performance measures for the initial design of BioSense, these estimates and measures did not reflect the implementation of its redesigned program. CDC subsequently developed a draft plan for the redesigned program that described high-level cost and timeline estimates; however, the estimates are not reliable, and the plan did not include performance measures. According to best practices, cost estimates should be well-documented, comprehensive and accurate, and must be credible before they can be considered to be reliable. However, CDC's cost estimates for the redesigned program are not reliable because they are only partially documented, are not comprehensive and accurate, and therefore are not credible. Best practices for reliable timeline estimates include the identification of resources to complete each task, establishment of a critical path, and analysis of risks to the schedule. However, the agency has not implemented these practices, resulting in timelines for the redesigned program that are not reliable. The Office of Management and Budget directs agencies to define outcome-based performance measures to gauge program results early enough for stakeholder review, and industry experts describe the need for stakeholder input in developing performance measures in order to monitor performance. While CDC established performance measures and benchmarks for the initial implementation of the BioSense program, it has not yet developed outcome-based performance measures to monitor the progress of the redesigned program and does not intend to complete their development until the end of 2009. Until program officials develop reliable cost and timeline estimates and outcome-based performance measures for the redesigned BioSense program, they will lack key components needed to effectively manage the program, increasing the risk that the agency will perpetuate weaknesses identified in its initial implementation of the program and related system.

Recommendations for Executive Action

Status: Closed - Implemented

Comments: In January 2013, CDC provided updated cost estimates for the redesigned BioSense program. These estimates were well documented and comprehensive and accurate, and met the cost estimation guide's best practice criteria for credibility. For example, for each cost element, a description of the element, the data source, and the estimating methodology was identified. Additionally, a risk analysis was conducted that quantified the risks, and costs were estimated by an independent group outside the acquiring organization to determine whether other estimating methods produced similar results. By developing well-documented, comprehensive and accurate, and credible cost data, CDC has taken steps that should improve its ability to estimate and more effectively track the cost of implementing BioSense, and to more effectively manage the development and implementation of the redesigned program and its related systems.

Recommendation: To ensure that CDC defines reliable plans for effectively managing the development and implementation of the redesigned BioSense program and its related system, the Director of CDC should instruct the Director of the National Center for Public Health Informatics to develop reliable cost estimates for the program that are well-documented, comprehensive and accurate, and credible while in the planning phase of the redesigned BioSense program.

Agency Affected: Department of Health and Human Services: Public Health Service: Centers for Disease Control

Status: Closed - Implemented

Comments: In September 2012, CDC provided GAO updated timeline estimates for the redesigned BioSense program that were complete and met GAO's cost estimation best practice criteria for reliability. Specifically, the estimates identified the resources needed to accomplish strategic activities and the schedule's critical path, and included an analysis of risks to the schedule. By taking steps to establish complete and reliable timeline estimates, CDC officials better defined plans to help effectively manage the development and implementation of the redesigned BioSense program and its related systems.

Recommendation: To ensure that CDC defines reliable plans for effectively managing the development and implementation of the redesigned BioSense program and its related system, the Director of CDC should instruct the Director of the National Center for Public Health Informatics to develop reliable timeline estimates for implementing the program while in the planning phase of the redesigned BioSense program.

Agency Affected: Department of Health and Human Services: Public Health Service: Centers for Disease Control

Status: Closed - Implemented

Comments: In August 2012, CDC released an evaluation plan for the program that contained outcome-based performance measures for BioSense that were developed with input from stakeholders. Specifically, the plan described measures related to user diversity, system usage, and the number of jurisdictions providing data for analysis by the system, among others. These measures are intended to be used to assess the extent to which the implementation of the system supports the program's progress toward meeting its goals. For example, program officials intend to measure progress toward the program's goal of regional collaboration by monitoring the number of jurisdictions electing to provide data to CDC for analysis by BioSense. Additionally, officials intend to gauge progress toward the goal of promoting a workforce competency to use public health surveillance tools by monitoring the diversity of the users and the number of jurisdictions using the system during public health events. Further, program officials engaged BioSense stakeholders when defining these measures and milestones for meeting them. Specifically, representatives from entities such as hospitals, state and local public health jurisdictions, and other federal agencies that play a role in delivering health care and public health provided input by participating in an expert panel and periodic collaboration meetings. BioSense program officials considered and incorporated their input while developing the program's evaluation plan and performance measures. For example, in expert panel meetings, stakeholders expressed the need for emergency department data, and program officials included in the performance measures a metric to indicate the percentage of emergency departments participating in the program. By engaging stakeholders in defining outcome-based performance measures that reflect the goals of the program, program officials should be better positioned to effectively gauge the success of the system's performance toward meeting CDC's and its federal, state, and local partners' goals and needs for improving abilities to respond to public health events.

Recommendation: To ensure that CDC defines reliable plans for effectively managing the development and implementation of the redesigned BioSense program and its related system, the Director of CDC should, with stakeholder input, instruct the Director of the National Center for Public Health Informatics to develop outcome-based performance measures that address all phases of the program and that focus the success of the program on CDC's federal, state, and local partners' use of the system for responding to public health events while in the planning phase of the redesigned BioSense program.

Agency Affected: Department of Health and Human Services: Public Health Service: Centers for Disease Control