Additional Materials:

Contact:

The Department of Veterans Affairs (VA) provides medical care, disability compensation, and vocational rehabilitation to veterans. The Veterans Health Administration (VHA)--a component of VA--provides care to over 5 million patients in more than 1,500 facilities. VHA relies on an outpatient scheduling system that is over 25 years old. In 2000, VHA began the Scheduling Replacement Project to modernize this system as part of a larger departmentwide modernization effort called HealtheVet. However, in February 2009, VA terminated a key contract supporting the project. GAO was asked to (1) determine the status of the Scheduling Replacement Project, (2) determine the effectiveness of VA's management and oversight of the project, and (3) assess the impact of the project on VA's overall implementation of its HealtheVet initiative. To do so, GAO reviewed project documentation and interviewed VA and contractor officials.

After spending an estimated $127 million over 9 years on its outpatient scheduling system project, VA has not implemented any of the planned system's capabilities and is essentially starting over. Of the total amount, $62 million was expended for, among other things, project planning, management support, a development environment, and equipment. In addition, the department paid an estimated $65 million to the contractor selected to develop the replacement scheduling application. However, the application software had a large number of defects that VA and the contractor could not resolve. As a result, the department terminated the contract, determined that the system could not be deployed, and officially ended the Scheduling Replacement Project on September 30, 2009. VA began a new initiative that it refers to as HealtheVet Scheduling on October 1, 2009. As of April 2010, the department's efforts on this new initiative had largely consisted of evaluating whether to buy or custom build a new scheduling application. VA's efforts to successfully complete the Scheduling Replacement Project were hindered by weaknesses in several key project management disciplines and a lack of effective oversight that, if not addressed, could undermine the department's second effort to replace its scheduling system: (1) VA did not adequately plan its acquisition of the scheduling application and did not obtain the benefits of competition. (2) VA did not ensure requirements were complete and sufficiently detailed to guide development of the scheduling system. (3) VA performed system tests concurrently, increasing the risk that the system would not perform as intended, and did not always follow its own guidance, leading to software passing through the testing process with unaddressed critical defects. (4) VA's project progress and status reports were not reliable, and included data that provided inconsistent views of project performance. (5) VA did not effectively identify, mitigate, and communicate project risks due to, among other things, staff members' reluctance to raise issues to the department's leadership. (6) VA's various oversight boards had responsibility for overseeing the Scheduling Replacement Project; however, they did not take corrective actions despite the department becoming aware of significant issues. The impact of the scheduling project on the HealtheVet initiative cannot yet be determined because VA has not developed a comprehensive plan for HealtheVet that, among other things, documents the dependencies among the projects that comprise the initiative. VA officials stated that the department plans to document the interdependencies, project milestones, and deliverables in an integrated master schedule as part of a project management plan that is expected to be completed by June 2010. In the absence of such a plan, the impact of the scheduling project's failure on the HealtheVet program is uncertain. Secretary of Veterans Affairs direct the Chief Information Officer to take six actions to improve key processes, including acquisition management, system testing, and progress reporting, which are essential to the department's second outpatient scheduling system effort. In written comments on a draft of this report, VA generally concurred with GAO's recommendations and described actions to address them.

Recommendations for Executive Action

Status: Closed - Implemented

Comments: In written comments on our report, VA concurred with this recommendation and stated that it would work closely with contracting officers to ensure future acquisition plans clearly identify an acquisition strategy that promotes full and open competition. The department subsequently issued a request for proposals to industry in November 2014, for the Medical Appointment Scheduling System (MASS) project that was informed by a request for information and market analysis. VA prepared an acquisition strategy for MASS that was intended to promote full and open competition and, in January 2015, the department received multiple proposals. VA subsequently awarded a contract for the MASS project in August 2015. As a result, VA has taken actions to ensure that its effort to replace its scheduling system will benefit from activities taken to promote full and open competition that were not taken for previous scheduling replacement efforts.

Recommendation: To enhance VA's effort to successfully fulfill its forthcoming plans for the outpatient scheduling system replacement project and the HealtheVet program, the Secretary of Veterans Affairs should direct the Chief Information Officer (CIO) to ensure acquisition plans document how competition will be sought, promoted, and sustained or identify the basis of authority for not using full and open competition.

Agency Affected: Department of Veterans Affairs

Status: Open

Comments: In written comments on our report, VA concurred with this recommendation and in August 2014, identified initial actions the department had taken in response. Specifically, as part of its plans to issue a request for proposals to acquire a replacement scheduling system under its Medical Appointment Scheduling System (MASS) project, VA developed a Business Requirements Document that defines its specific business needs, capabilities, features, and constraints. Additionally, the department reported that it intends to manage and document requirements using processes supported by a Web-based tool called Rational Doors. In August 2015, VA's Office of Acquisition, Logistics, and Construction awarded a contract for the MASS project. However, in April 2016, the department paused the MASS project in order to evaluate an option to enhance its legacy scheduling system. Additionally, as of May 2016, the department had not developed a requirements management plan for the MASS project. Thus, the MASS project has not yet reached the point where the effectiveness of the requirements management activities we recommended can be assessed.

Recommendation: To enhance VA's effort to successfully fulfill its forthcoming plans for the outpatient scheduling system replacement project and the HealtheVet program, the Secretary of Veterans Affairs should direct the CIO to ensure implementation of a requirements management plan that reflects leading practices for requirements development and management. Specifically, implementation of the plan should include analyzing requirements to ensure they are complete, verifiable, and sufficiently detailed to guide development, and maintaining requirements traceability from high-level operational requirements through detailed low-level requirements to test cases.

Agency Affected: Department of Veterans Affairs

Status: Open

Comments: In written comments on our report, VA concurred with this recommendation and stated that testing was managed using documented, repeatable processes that are included in the department's ProPath Web-based tool. According to the Acting Deputy Chief Information Officer for Product Development, the Medical Appointment Scheduling System (MASS) project is expected to incorporate Agile software development practices, including the use of incremental testing. In August 2015, the department awarded a contract for the MASS project that included task orders for the development of test plans. However, in April 2016, the department paused the MASS project in order to evaluate an option to enhance its legacy scheduling system and, as of May 2016, had not developed a test plan for MASS. Thus, the MASS project has not yet reached the point where adherence to the department's system testing guidance can be assessed.

Recommendation: To enhance VA's effort to successfully fulfill its forthcoming plans for the outpatient scheduling system replacement project and the HealtheVet program, the Secretary of Veterans Affairs should direct the CIO to adhere to the department's guidance for system testing including (1) performing testing incrementally and (2) resolving defects of average and above severity prior to proceeding to subsequent stages of testing.

Agency Affected: Department of Veterans Affairs

Status: Closed - Not Implemented

Comments: In written comments on our report, the Department of Veterans Affairs stated that it "concurred in principle" with our recommendation and noted that its Project Management Accountability System (PMAS) requires monthly analysis and reporting of project performance. However, PMAS project status reporting does not include EVM data.

Recommendation: To enhance VA's effort to successfully fulfill its forthcoming plans for the outpatient scheduling system replacement project and the HealtheVet program, the Secretary of Veterans Affairs should direct the CIO to ensure effective implementation of earned value management (EVM) by making certain that the: (1) EVM reporting systems for the scheduling project are certified for compliance with American National Standards Institute/Electronic Industries Alliance (ANSI/EIA) Standard 748 and data resulting from the systems are reliable; (2) project status reports based on EVM data are reliable in their portrayal of the project's cumulative and current cost and schedule performance; and (3) officials responsible for managing and overseeing the project use earned value data as an input to their decision-making processes.

Agency Affected: Department of Veterans Affairs

Status: Closed - Implemented

Comments: The Department of Veterans Affairs concurred with our recommendation. The department created a risk register for its Medical Appointment Scheduling System (MASS) project that identifies risks as well as their probability and impact. In addition, the department instituted the use of its Project Management Accountability System and its ProPath Web-based tool to help mitigate risks related to the department's effort to replace its scheduling system.

Recommendation: To enhance VA's effort to successfully fulfill its forthcoming plans for the outpatient scheduling system replacement project and the HealtheVet program, the Secretary of Veterans Affairs should direct the CIO to identify risks related to the scheduling project moving forward and prepare plans and strategies to mitigate them.

Agency Affected: Department of Veterans Affairs

Status: Open

Comments: The Department of Veterans Affairs concurred with this recommendation and identified various actions it has taken in response. Specifically, the agency instituted its Project Management Accountability System (PMAS) as a means of managing and overseeing IT projects. PMAS includes policies on actions the department is to take when development deficiencies for a project are identified, including for the Medical Appointment Scheduling System (MASS) project. As part of this project management and oversight approach, in June 2014, the department completed a milestone 0 review that formally began planning for the MASS project. Subsequently, in September 2014, VA completed a milestone 1 review, which formally initiated the MASS project. These reviews included information on the project's budget, schedule, risks, and dependencies on other projects. In addition, the agency has begun to use the ProPath Web-based tool to document and store key systems development life-cycle processes and their associated documentation; further, the agency awarded a contract for the MASS project in August 2015. However, in April 2016, the department paused the MASS project in order to evaluate an option to enhance its legacy scheduling system. In May 2016, VA reported that the MASS project will adhere to the department's new Veteran-focused Intake Process, which is intended to ensure oversight, accountability, and traceability of all program activity. Nevertheless, the MASS project has not yet reached the point where the effectiveness of VA's oversight can be assessed.

Recommendation: To enhance VA's effort to successfully fulfill its forthcoming plans for the outpatient scheduling system replacement project and the HealtheVet program, the Secretary of Veterans Affairs should direct the CIO to ensure that the policies and procedures VA is establishing to provide meaningful program oversight are effectively executed and that they include (1) robust collection methods for information on project costs, benefits, schedule, risk assessments, performance metrics, and system functionality to support executive decision making; (2) the establishment of reporting mechanisms to provide this information in a timely manner to department IT oversight control boards; and (3) defined criteria and documented policies on actions the department will take when development deficiencies for a project are identified.