The Federal Bureau of Investigation's
Management of the Trilogy Information Technology Modernization Project

Audit Report No. 05-07
February 2005
Office of the Inspector General

Findings and Recommendations

Finding: The Schedule, Cost, Technical, and PerformanceBaselines of the Trilogy Project

While Trilogy has greatly improved the FBI’s information technology infrastructure, the critical VCF application remains incomplete and will not result in the required modernization of the FBI’s case management system in the foreseeable future. The FBI plans to pilot test a prototype VCF by early 2005, but this test and subsequent evaluation will only involve workflow processes in conjunction with the existing ACS system, rather than a fully functional case management system. On a separate track, the FBI is leading an interagency effort to develop a Federal Investigative Case Management System (FICMS) framework, which is intended to lead to the development of fully functional investigative case management systems for the Departments of Justice and Homeland Security. However, the FBI will not have schedule and cost estimates for FICMS and the resulting case management system until April 2005 at the earliest, when it awards the FICMS contract. While the completed infrastructure components of Trilogy have provided the FBI with a sorely needed IT upgrade, the continuing lack of an effective case management system hinders the FBI’s capability to perform its critical national security mission.

Additionally, the Trilogy project has been plagued throughout its development with missed deadlines and rising costs. While events affecting both the mission and operations of the FBI resulted in project modifications, had the Trilogy contracts included fully established requirements and firm completion milestones, the adverse effects of such changes could have been mitigated. Recently, the FBI has restructured and tightened its IT management and now intends to more rigorously manage its pursuit of a new solution to its case management requirements.

Need for Trilogy

Both the FBI’s IT infrastructure and its case management system were in dire need of modernization at the time of Trilogy’s initiation. Both aspects of the project are extremely important and interrelated —
without the upgraded infrastructure a modern, fully functional case management system with information-sharing capabilities could not be implemented. At the same time, without an effective case management system the FBI cannot identify and capitalize on all of the information in its possession. With the completion of Trilogy’s infrastructure components, the FBI has the ability to support an enhanced case management system. However, without a modern, fully functional case management system in place, agents and analysts largely rely on paper case files and an existing but obsolete case support system. Consequently, agents and analysts are at a severe disadvantage in performing their duties.

Infrastructure Components

In April 2004, the FBI completed the infrastructure components of Trilogy. However, although Congress funded an accelerated schedule for completing these components, the FBI only met the original target date. As described in the Introduction section of this report, for much of the Trilogy project’s history the FBI had never established a stable schedule for development and deployment of the infrastructure components. Beginning in 2002, the FBI’s estimated dates for completing the infrastructure began to fluctuate and were revised repeatedly. At one point, the FBI moved up the target completion date for deploying the Trilogy infrastructure from May 2004 to June 2003 in light of the September 11 attacks and the need to enhance the FBI’s IT infrastructure as rapidly as possible. Subsequently, the FBI said the infrastructure deployment would be completed by December 2002. After receiving additional funding from Congress in FY 2002 to accelerate the project, the FBI again revised the completion date for the infrastructure to July 2002.

The plan to accelerate Trilogy involved deploying the IPC/TNC infrastructure enhancements in three phases. The first phase, called
Fast Track, included the installation of Trilogy hardware in FBI field offices and the larger resident agencies. The Fast Track deployment consisted of new network printers, color scanners, local area network upgrades, desktop workstations, and office automation software. FBI officials reported that by the end of December 2002, all field offices had Fast Track completed. This included 22,251 workstations, 3,408 printers, 1,463 scanners, and 475 servers.

Following the completion of Fast Track, the FBI initiated the next phase of the hardware deployment, Extended Fast Track. Completed in March 2003, Extended Fast Track: 1) installed servers and other network components at field office and resident agency sites, and
2) deployed the hardware included under Fast Track to additional resident agency sites that were not included in the first phase. The FBI also intended Extended Fast Track to correct any shortfalls in the distribution of hardware to the field offices that occurred in the original Fast Track deployment. During this phase, the FBI also deployed a wide area network (WAN) for 593 FBI sites. The WAN was certified and accredited to meet the security requirements for operating within the FBI.

The final phase of the infrastructure deployment, called
Full Site Capability, represented the complete infrastructure upgrade. This phase provided WAN connectivity together with new encryption devices, new operating systems and servers, and new and improved
e-mail capability. However, completion of this phase involved extensive contract renegotiations with Computer Sciences Corporation (CSC), into which DynCorp had merged, and increased efforts on the FBI’s part to manage the project and encourage CSC to complete the Full Site Capability. The final year of the infrastructure component’s evolution is discussed below.

The milestone for completing the infrastructure components slipped from the initial date of July 2002 to March 2003. On March 28, 2003, CSC completed the Local Area Network (LAN) for Trilogy. In March 2003, the FBI Director reported to Congress that the Trilogy LAN — with increased bandwidth and three layers of security — had been deployed to 622 sites.

In April 2003, the FBI and CSC agreed to a statement of work for the remaining infrastructure components of Trilogy, including servers, upgraded software, e-mail capability, and other computer hardware, with final engineering change proposals and a completion date of October 31, 2003. In August 2003, CSC informed the FBI that the October 2003 completion date would slip another two months to December 2003. In October 2003, CSC and the FBI agreed that the December 2003 date again would slip. In November 2003, the GSA’s FEDSIM announced that CSC failed to meet the deadline for completing work on the infrastructure portions of Trilogy that are required to support the user applications, including the VCF.

On December 4, 2003, CSC signed a commitment letter agreeing to complete the infrastructure portions of Trilogy by April 30, 2004, for an additional $22.9 million, including an award fee of over $4 million.11 The FBI covered these additional costs by reprogramming funds from other FBI appropriations. In January 2004, the FBI converted the agreement with CSC to a revised statement of work providing for loss of the award fee if CSC did not meet the April 30, 2004, deadline. In addition, the revised statement of work provided for a cost-sharing rate of 50 percent if any work remained after April 30.

In April 2004, CSC installed the final infrastructure pieces in FBI field offices needed to use the previously deployed WAN, thereby completing the infrastructure portion of the project and meeting its contractual requirements. In the end, CSC completed the infrastructure component by May 2004 – the FBI’s original target
date – but missed the completion date under the accelerated schedule funded by Congress by some 22 months. The total expected costs for the infrastructure components of Trilogy increased from $238.6 million to $339.8 million over the course of the project.

User Applications and the Virtual Case File

The design of and schedule for the UAC portion of Trilogy were substantially modified after the September 11 attacks. The most significant design change was eliminating the web-enablement of ACS and instead developing an enterprise-wide solution to replace ACS, an obsolete and inconsistently utilized case information system. The replacement for the web-enablement of ACS was to be deployed in two phases under an accelerated plan: delivery one and delivery two. A third delivery was added in March 2003.

Delivery one of the UAC was intended to consist of a new application known as the Virtual Case File (VCF). The VCF was intended to be a completely new investigative case management system with data migrated from the ACS. The VCF was to serve as the backbone of the FBI’s information systems, replacing the FBI’s paper case files with electronic files. The first delivery of VCF was targeted for completion in December 2003. The second and third deliveries, which were intended to upgrade and add additional investigative applications to the VCF, were targeted for completion in June 2004.

At the outset of the Trilogy project, the UAC was intended to replace each of the following five primary user applications:

the Criminal Law Enforcement Application (CLEA), a repository of criminal investigation data;

the Integrated Intelligence Information Application (IIIA), which supports counterintelligence and counterterrorism investigations by enabling the collection, collation, analysis, and dissemination of intelligence; and

the Telephone Application (TA), which provides a central repository for telephone data obtained from investigations.

According to one VCF project manager, the plan to replace each of the FBI’s five primary investigative applications was not based on an objective evaluation of the operational needs of the organization, but rather on an assumption made by the FBI’s IT managers at the time that replacing these applications would have the greatest benefit for agents and analysts.12 The FBI refined the VCF concept through Joint Application Development (JAD) sessions held between January and June 2002. The JAD sessions brought together FBI representatives to determine what applications were needed to support the case management and information requirements of FBI agents, analysts, and support personnel; UAC contractor representatives to determine what applications could be created; and infrastructure contractor representatives to ensure that the applications could be supported by the groundwork that was being developed.

The VCF plan that resulted from these JAD sessions in 2002 rejected the previous plan to replace the five separate investigative applications in favor of developing an entirely new electronic workflow with systems that are integrated into one process. The VCF concept
not only would change where the data for case files is stored, but also would create an entirely new environment in which agents, analysts, and support personnel operate. This workflow would be based on the information required to create case files and would combine aspects of ACS, CLEA, and TA applications into one application. After these sessions, the FBI removed IntelPlus from the planned VCF to avoid redundancy with the current version of the Information Data Warehouse (IDW) project and the revamped VCF.13

Implementation of the VCF would require a complete change in how agents establish case files. Rather than creating paper records as they currently do, agents would be required to input case information into the VCF electronically to complete a series of data fields. The data fields would be tagged so that the information entered into the VCF is consistent throughout the system. These electronic "case files" would be the sole case files utilized by the FBI, so that only one set of data would exist for a case and only one search within the VCF would be required to locate and view case-related information.

VCF case files would enable a wide variety of functionalities. For example, when a "parent" case file is established by one FBI field office and an action relating to that case is taken by another field office (such as a fraud case in New York requiring a summons being delivered in Philadelphia), a "child" to the original case file can document all of the activities related to the parent case. This capability would allow the entire case history to be traceable from initiation through closure.

In addition, the VCF was intended to include a multimedia capability that would rectify a longstanding information-sharing limitation within the FBI. Agents would be able to scan documents, photographs, and other electronic media into the case file. This capability would allow evidence and other case-related information to be shared among agents working on a case FBI-wide without the need to exchange physical copies of the information. Also, under the case structure established in the first delivery of the VCF, the Reports on Investigative Activity (RIA) component of a case eventually would
allow data comparisons and correlations to be made between cases in order to "connect the dots."14

According to a former VCF project manager at the FBI, the VCF would also incorporate additional applications to aid agents’ work. For example, because ACS does not offer statistical reporting capabilities, agents have had to perform such tasks using a separate application outside of ACS. The VCF would incorporate the features of the Integrated Statistical Reporting and Analysis Application, which maintains case-related statistics such as the number of investigative leads that result in arrests and convictions. The VCF also would allow agents to request funding for cases rather than to have to enter requests through a separate financial management system.

Current Schedule for User Applications Component

The VCF contractor, SAIC, delivered the first of three planned system deliveries for the VCF in December 2003. However, the application was not fully functional and the FBI did not accept the product. FBI officials stated that 17 issues of concern pertaining to the functionality and basic design requirements of the VCF needed to be resolved before the application could be deployed. According to FBI personnel working on the resolution of these problems, the 17 issues were corrected as of March 7, 2004.

However, significant work still remained on the VCF, including security aspects and records management issues. As a result, the FBI revised the VCF deployment schedule again. Rather than having the VCF implemented and enhanced in three deliveries, the FBI Director announced in June 2004 that development of the project would be split into two parallel tracks, "Initial Operational Capability" (IOC) and "Full Operational Capability" (FOC). Neither Track One, the IOC, nor Track Two, the FOC, will result in the deployment of a fully functional VCF with case management capabilities. As discussed in more detail below, the two tracks will test a limited VCF paperwork flow feature and continue to identify and refine user requirements that the FBI believes will eventually lead to the development of a new case management system through the Federal Investigative Case Management System (FICMS) effort to replace the stalled VCF project.

On July 30, 2004, the FBI signed a contract modification with SAIC to work on Track One. By December 31, 2004, SAIC provided the FBI a "proof-of-concept" or prototype VCF. This prototype is being used by agents and analysts in a test field office to demonstrate an electronic workflow process in ongoing investigations. Over a 6-week pilot test period scheduled between January and March 2005, the FBI’s New Orleans Field Office and the Baton Rouge Resident Agency are entering actual investigative lead and case data into the VCF. This information will be uploaded into the ACS to create paper case files upon electronic approval of the electronic information in the VCF.

Yet, the version of the VCF tested in Track One will not provide the FBI its goal of a paperless records management application or a system that will handle the vital investigative leads, evidence, and case management application. Rather, the goal of the Track One test phase is to determine the efficiency and effectiveness of the electronic workflow and document approval process — even while still using the antiquated ACS system. Due to its limited progress in developing the VCF, the FBI has determined that SAIC will not provide the previously planned second and third deliveries. These enhancements were intended to upgrade the VCF’s capabilities and provide a deployable VCF that would enable the FBI to efficiently manage its investigative cases and documents and allow agents and analysts to access and share investigative information much more effectively.

Track Two, the FOC, also will not result in the delivery of a fully functional VCF. Instead, Track Two represents an effort to reevaluate and identify new requirements for developing a functional case management system to replace the ACS. Under Track Two, the FBI is identifying requirements for a new system, including user activities and processes for creating and approving documents and managing investigative leads, evidence, and cases. As a part of Track Two, a separate contractor, The Aerospace Corporation, evaluated the initial VCF delivery to determine the extent to which the design can be of further use to the FBI.

In essence, under Track Two, the FBI will update system requirements for the future development of a case management system.15

The VCF effort that began in June 2001 has been unable to meet the FBI’s case management needs. Instead, the FBI has taken a new
approach for pursuing the development of a case management system. On September 14, 2004, the Departments of Justice and Homeland Security, in conjunction with the Office of Management and Budget, released a Request For Information for the FICMS (see Appendix 3). The FICMS is intended to result in a common platform in support of a multi-phased program to modernize investigative and intelligence processes within the federal government as a whole. On September 28, 2004, the Departments of Justice and Homeland Security — with the FBI acting as the executive agent — presented an overview of the FICMS to vendors in order to obtain information on existing or potential solutions for this multi-agency case management system.16 Nearly 100 vendors, including SAIC, attended the FBI’s presentation, which included a discussion of the challenges facing the FBI in implementing the VCF, as well as the requirements that have been identified in working on the VCF. The requirements include creating and managing electronic case files; assigning cases; integrating workflow and document management including the creation, review, collaboration, approval, storage and disposition of documents; evidence management; and records search and reporting. The FICMS overview also outlined security measures and labeling requirements for national security information.

The FBI told the OIG that it anticipates issuing a Request for Proposals by about January 2005 and awarding a contract by April 30, 2005, to develop the FICMS framework through a step-by-step modular approach. The FBI hopes that through this approach the project will be broken down into manageable components such as workflow, security, records management, evidence management, investigative leads management, and administration. Further, the system architecture will be developed to comply with the information-sharing requirements of Executive Order 13356.

However, until the FICMS framework is developed and an investigative case management system is developed and deployed, the FBI will continue to use the poorly functioning ACS to manage its cases. The FBI is unable to provide even a rough estimate of the schedule or costs for developing the FICMS framework and resulting system until it receives and evaluates the contractors’ proposals. In the meantime, FBI personnel will continue to use paper files and the limited capabilities of the antiquated ACS. Although the FBI emphasized that it has scanned a multitude of documents into its data warehouse, this document repository does not equate to the case management and information-sharing and searching capabilities envisioned for the VCF.

The Current VCF Application Will Not Meet the FBI's Needs

After committing over three years and $170 million — nearly $51 million more than initially estimated — the FBI thus far has not completed the original UAC portion of Trilogy, the critical VCF.

The urgent need within the FBI to create, organize, share, and analyze investigative leads and case files on an ongoing basis remains unmet. As of December 31, 2004, the VCF will only result in a test version of a new workflow system involving the capability to upload of documents into the ACS system. From that point, the FBI is further refining the user needs and more fully developing requirements for a future investigative case management system. The FBI is now counting on the interagency FICMS effort to eventually result in
a modern case management system primarily using off-the-shelf technology that was unavailable in 2001 when Trilogy’s development began.

According to the FBI’s CIO, about 80 percent of the FBI’s case management system requirements are consistent with other federal investigative agencies, so that a “flagship” platform such as that envisioned through FICMS is feasible. Further, the FBI CIO said that using existing IT solutions in developing a case management system under FICMS (commercial off-the-shelf systems or government off-the-shelf systems) should reduce its costs that will be shared by the participating agencies. The FBI expects to pay its share of FICMS developmental costs by reprogramming some of its IT appropriations. The extent to which the VCF development effort to date can be leveraged in developing an interagency case management platform is unknown at this point, because the FICMS is in its early conceptual stage. In our opinion, although the VCF effort helped the FBI define its user requirements, a successor case management system is unlikely to benefit substantially from the VCF from a technical or engineering standpoint due to: 1) advances in technology during the lengthy VCF developmental process, and 2) the FBI’s intended approach of adapting existing systems to provide the various components of the case management system.

Although the development of an interagency investigative case management system is desirable to enable information to be shared across agencies, Trilogy was supposed to provide case management and information-sharing capabilities for the FBI. While the FBI is looking toward developing a system through the FICMS effort to ultimately replace the largely unsuccessful VCF, the ability of the FBI to adequately compile, analyze, and share case information within its own organization will be delayed as the FICMS framework and resulting system are developed.

Problems in Trilogy's Development

We found that the delays and associated cost increases in the Trilogy project have resulted from several factors, including:

poorly defined and evolving design requirements,

contracting weaknesses,

IT investment management weaknesses,

lack of an Enterprise Architecture,

lack of management continuity and oversight,

unrealistic scheduling of tasks,

lack of adequate project integration, and

inadequate resolution of issues raised in reports on Trilogy.

The combination of these factors has resulted in a Trilogy project that received significant additional funding to accelerate completion, but that after three years has not been fully implemented. It now appears that the multi-agency FICMS effort, which currently is purely conceptual, will result in a replacement of the existing inadequate VCF as the FBI’s solution to meet its need for an investigative case management system. A discussion of the reasons for Trilogy’s schedule, cost, technical, and performance problems follows.

Poorly Defined and Slowly Evolving Design Requirements

One of the most significant problems with managing the schedule, cost, and technical aspects of the Trilogy project was the lack of a firm understanding of the design requirements by both the FBI and the contractors. Trilogy’s design requirements were ill-defined and evolving as the project progressed. In addition, certain events required the need to modify initial design concepts. For example, after the September 11 attacks the FBI recognized that the initial concept of simply modifying the old ACS would not serve the FBI well over the long run. The FBI then created plans for the VCF. Additionally, a need for broadened security requirements due to vulnerabilities identified in the Hanssen espionage case affected Trilogy’s development. According to one project manager, this recognition of the need to upgrade security caused more problems and delays for the full implementation of the infrastructure component.17

During the initial years of the project, the FBI had no firm design baseline or roadmap for Trilogy. According to one FBI Trilogy project manager, Trilogy’s scope grew by about 80 percent since initiation of the project. Such large changes in the requirements meant that the specific detailed guidance for the project was not established, and as a result a final cost and schedule was not established. The project manager stated that for future projects, requirements must be defined in specific detail, and all parties involved in such a project must be onboard with the requirements.

Another problem in managing Trilogy was the limited engineering capabilities of the FBI and the contractor in creating the VCF. In January 2004, the acting Chief Technology Officer brought in a group of engineering contractors to review technical and performance aspects of the VCF. This review examined the technical design decisions, reliability, performance, and resiliency of the system being developed by the contractor. The main objective of the group was to identify fundamental design flaws. The group’s decision paper cited 37 basic design flaws, including network, server, and storage infrastructure issues, operating system and software issues, application issues, and problems with the test plan. The lack of redundancy and resiliency in the system was seen as a major flaw because the design failed the basic "can of soda" test. This test simply shows that if a can of soda were to be spilled on one of the servers, causing it to fail, that server’s backup would also fail because it was located directly below and would also be damaged by the liquid.

The contractors performing the review stated that the VCF design was adversely affected by a lack of engineering expertise on the project. The group said that the FBI should have approached developing the Trilogy system from an engineering perspective rather than just by relying on what the system should achieve for the user. According to the contractors, this approach should have been implemented from the outset of the project so that the FBI could have acted as an educated consumer, directing more of what the contractor should have been implementing in the technical development of the VCF rather than just from a user perspective.

The current status of the VCF demonstrates that the lack of fully developed requirements for the project negatively affected schedule, cost, technical, and performance baselines. At this point in the project’s development, a limited-use application that will continue to create paper files is the most optimistic outcome of the current Trilogy user application contract. However, the technical and performance requirements for the future inter-agency case management system are yet to be fully defined.

Contracting Weaknesses

The FBI’s current and former CIOs told us that a primary reason for the schedule and cost problems associated with Trilogy was weak statements of work in the contracts. According to FBI IT and contract managers, the cost-plus-award-fee type of contract used for Trilogy:

did not require specific completion milestones,

did not include critical decision review points, and

did not provide for penalties if the milestones were not met.

Under cost-plus-award-fee contracts, contractors are only required to make their best effort to complete the project. Furthermore, if the FBI does not provide reimbursement for the contractors’ costs, under these agreements the contractors can cease work. Consequently, in the view of the FBI managers with whom we spoke, the FBI was largely at the mercy of the contractors. A Trilogy project manager compiling lessons learned from the project told us that the FBI should require all future IT projects to operate solely with
fixed-price contracts that contain award fees.18 He stated that this type of contract would greatly enable the FBI to achieve the results being sought without relying too heavily on contractors. However, in order to use such contracts, the manager noted, the FBI must have much more fully defined requirements for a project.

Because the FBI wanted to award the Trilogy contracts quickly and did not have clearly defined requirements, it used the cost-plus-award-fee contract vehicle. While the contracting process was expedited, the lack of well-defined requirements severely affected the timeliness and cost of the implementation of the Trilogy project.

IT Investment Management Weaknesses

At Trilogy’s inception and over much of its life, the FBI’s IT Investment Management (ITIM) process was not well-developed, as described in the OIG’s December 2002 audit report, The Federal Bureau of Investigation’s Management of Information Technology Investments. That audit found that while the FBI had started centralizing its project management structure, it still needed to integrate its ITIM process with a standardized project management methodology. Specifically, the report stated that project management was not consistently followed by IT project managers and policies and procedures were not developed for management oversight of IT projects. This lack of oversight included a lack of project management plans that would include cost and schedule controls.

The report included a brief case study of Trilogy, and the lack of a mature ITIM process was found to contribute to the missed milestones and uncertainties associated with Trilogy. The report stated that most of Trilogy’s development had been managed in a "stovepipe," and as a result FBI personnel not involved in the management of Trilogy had little knowledge of the project’s status and progress. Additionally, there was little coordination among Trilogy management and contract specialists from the Finance Division or the Information Resources Division unit responsible for procurement of non-Trilogy IT needs. Finally, the philosophy employed in implementing Trilogy, according to the original FITUP plan, was "to get 80% of what is needed into the field now rather than 97% later. Then we can proceed in an orderly fashion to move toward 100% in the
future." In essence, the FBI took risks to expedite Trilogy’s implementation to the field, and that approach failed because the management processes requiring the creation of baselines for Trilogy were simply not in place.

Had the FBI developed a mature ITIM process — and the schedule, cost, technical, and performance baselines for the project been fully developed through the ITIM process — the Trilogy project likely could have been completed more efficiently and timely. The development of a mature ITIM process is ongoing within the FBI, and most of the recommendations of the OIG’s report on this subject have been implemented. However, absent a mature ITIM process, all FBI IT investment efforts are at a risk for the significant developmental problems experienced by Trilogy.

Lack of an Enterprise Architecture

An Enterprise Architecture provides an organization with a blueprint to more effectively manage its current and future IT infrastructure and applications. As stated in the Government
Accountability Office’s (GAO) report Information Technology: FBI Needs an Enterprise Architecture to Guide Its Modernization Activities, issued in September 2003, the development, maintenance, and implementation of Enterprise Architectures are recognized hallmarks of successful public and private organizations. The GAO reported that the FBI does not have an Enterprise Architecture, although it began developing one in early 2000. The GAO also found that the FBI lacks the management structures and processes to effectively develop, maintain, and implement an Enterprise Architecture.

Additionally, the OIG’s December 2002 audit report entitled FBI’s Management of Information Technology Investments recommended that the FBI continue its efforts to establish a comprehensive Enterprise Architecture. The report also recommended that the FBI develop and implement a specific plan to integrate the ITIM and Enterprise Architecture processes. While the FBI agreed to develop a comprehensive Enterprise Architecture, this recommendation has not been fully implemented. The FBI has contracted for an Enterprise Architecture to be completed by September 2005. Without a complete Enterprise Architecture, the FBI’s systems are not defined. As a result, in the Trilogy project the FBI needed to conduct reverse engineering to
identify existing IT capabilities before developing the infrastructure and user applications requirements.

Lack of Management Continuity and Oversight

Turnover in key positions has inhibited the FBI’s ability to manage and oversee the Trilogy project. Since November 2001, 15 different key IT managers have been involved with the Trilogy project, including 5 CIOs or Acting CIOs and 10 individuals serving as project managers for various aspects of Trilogy. This lack of continuity among IT managers contributed to the lack of effective and timely implementation of the Trilogy project.

According to contractor personnel who are advising the FBI on Trilogy, the FBI has suffered from a lack of engineering expertise, process weaknesses, and decision making by committees instead of knowledgeable individuals. In the opinion of the contractors with whom we spoke, weak government contract management was more of the problem with Trilogy than the terms of the contracts.

In addition to the lack of consistent management, the processes
used to manage Trilogy were also inadequate. For example, the FBI’s ability to adequately track Trilogy costs was questioned by a March 3, 2004, FBI inspection report. The inspection review was conducted to ensure that transactions for the Trilogy project were documented and recorded accurately, and to determine whether the financial management of the Trilogy project was in compliance with federal regulations and FBI policies and procedures.

The inspectors found that the FBI’s Financial Management System did not capture detailed Trilogy-related expenditures, while numerous entities tracked and monitored specific segments of the operation. Overall, Trilogy-related financial records were fragmented
and decentralized with no single point of accountability. The FBI’s Project Management Office did not implement a centralized budget, accounting, and procurement structure to ensure global financial management oversight.

The report also found that an individual functioning in an advisory role to the FBI for increasing funding of the various contracts associated with Trilogy worked for a contractor that provided IT services on Trilogy. As a result, the inspectors found a conflict of interest in decision-making regarding the Trilogy project. Additionally, although the contractor maintained detailed records for vendor invoicing as part of her bookkeeping duties, she did not have access to overall FBI financial data and therefore was unable to provide the FBI with a complete picture of Trilogy costs.

Finally, the FBI internal report stated that the Budget Unit of the FBI’s Information Resources Division was not reconciling or updating portions of the Trilogy tracking report, which resulted in discrepancies in the dollar amounts reported to management. During the review, the inspectors gave the information to the Budget Unit in order to coordinate a reconciliation of the Trilogy project’s funding.

The FBI said it has resolved these issues, but the cost-reporting problems that occurred demonstrate another area in which a thorough, regimented program management framework would have allowed the FBI to better capture and monitor expenditures and funding for the Trilogy project.

We interviewed officials in the FBI, the Department, and FEDSIM, many of whom said that the FBI recently has improved its management and oversight of Trilogy and of IT in general. The FBI appears to have hired capable IT managers from other federal agencies and private industry, including the current CIO and several key project management personnel. Officials within both the Department and the FBI are optimistic that the FBI’s current IT management team has the talent to solve the FBI’s seemingly intractable IT problems. That said, we believe it is essential for the FBI to maintain continuity in its management of Trilogy.

Unrealistic Scheduling of Tasks

Along with the lack of firm milestones in the Trilogy contracts, the scheduled completion dates for individual project components were unrealistic. According to an FBI official monitoring the development of the Trilogy infrastructure, CSC had problems producing an appropriate resource-driven work schedule. Until the FBI became more active in examining the scheduling of the project, the FBI accepted the project’s schedules as presented by the contractor. This acceptance began to shift when the FBI’s scheduler worked with the contractor to establish a realistic resource-driven work schedule for completing the infrastructure components. At that point, the contractor disagreed
with the resulting schedules, because the schedules showed that the full implementation of the project exceeded the proposed completion date of the project. According to the FBI official, the schedule showed a completion date in 2004, not the more optimistic October 2003 date that the contractor desired. When it became apparent that the infrastructure would not be implemented in October 2003, contract renegotiations established a more realistic completion date of April 30, 2004. The infrastructure components were completed in April 2004 in accordance with the revised schedule.

According to the FBI’s Project Management Office scheduler, the contractor for the User Applications Component (UAC), SAIC, used a scheduling tool for the development of the VCF with which the FBI was unfamiliar. As a result, the FBI was unable to determine if the assumptions within the schedule were reasonable and whether the implications on the schedule were adequately reflected. Thus, the FBI was unable to validate the contractor’s schedule for completing the project.19

In our view, unrealistic scheduling of project tasks led to a series of raised expectations, followed by frustration when the completion estimates were missed. Additionally, the FBI’s lack of familiarity with the contractor’s scheduling system for the UAC may have limited the FBI’s ability to quickly recognize the extent of schedule slippages and take steps to mitigate them. Standardizing the scheduling process among contractors would facilitate the FBI’s ability to recognize potential problems with project milestone dates.

Lack of Adequate Project Integration

Despite the use of two contractors to provide three major project components, the FBI did not hire a professional project integrator to manage contractor interfaces and take responsibility for the overall integrity of the final product until the end of 2003. According to FBI IT managers, FBI officials performed the project integrator function even though they had no experience performing such a role. Although FBI and Department officials stated that the Department required the FBI to perform project integration duties without contractor support, the expertise to adequately perform this function did not exist within the FBI. The problems involved with the scheduling of the project would have been more apparent to the FBI had proper project integration efforts taken place. A professional project integrator could have coordinated the scheduling of the infrastructure with the VCF implementation. Any delays in completing the infrastructure
component would have pushed back the full implementation of the VCF, and project costs could rise. Yet the FBI was not fully aware that the infrastructure would not meet its target date until August 2003. Additionally, until December 2003, the FBI was unaware that significant changes to the VCF were needed to achieve the desired performance.

If monitoring the scheduling of the project had been a priority, the FBI could have taken more timely action to effectively address Trilogy’s problems. At the end of 2003 — well over two years into the project — the FBI hired a contractor to perform these project integration duties when it became apparent that a professional project integrator was needed to effectively complete the project. According to the Quarterly Congressional Report on Trilogy for the period ending April 30, 2004, the FBI has initiated a termination of the integrator’s contract due to cost considerations. As of the reporting period, the integrator had received over $2.8 million.

Inadequate Resolution of Issues Raised in Reports

The FBI’s management of its IT, including the Trilogy project, was the focus of several reports issued both by components within the FBI and external reviewing entities, including the OIG. These reports are summarized in Appendix 2 of this report. The following discussion of the internal reports covering Trilogy demonstrates that the FBI took inadequate actions to resolve the findings of the reports. Because of the lack of resolution, many of these issues have remained throughout the course of the project.

Within a matter of months after the initiation of the Trilogy project, the FBI recognized significant issues that needed resolution. The internal reports issued by the FBI’s Inspection Division, CJIS Division, and consultants identified a lack of a single project manager, undocumented requirements, and a baseline that was not frozen. The CJIS Division’s assessment concluded that:

The predominant area of concern for the Trilogy program appears to be an overall lack of consistency. From baselines to technical direction, from clear lines of authority to a single accountable program manager, the message is clear – without consistency, it is increasingly difficult to apply standard contract analysis tools and methodologies to gain helpful insight into contract status and progresses.

Additionally, the report stated that "until requirements are documented and a baseline is established, it is difficult at best to effectively gauge the many aspects of project management that should be in place on a program of this maturity."

Based on its own reports, the FBI was aware of the risks that faced the Trilogy project. While FBI management eventually hired a project manager to oversee the project — a recommendation made in all of the reports — the process of defining requirements and baselines for the VCF continues, more than two years after these internal reports were issued. Although the Hanssen espionage case and the difficulties experienced in retrieving documents for the Oklahoma City bombing case (events that occurred after these reports were released) resulted in additional security and information-sharing requirements, the difficulties of incorporating the changes would have been greatly minimized had the FBI established project requirements and baselines in advance.

The FBI's Current and Future IT Needs

As discussed in the Background section of this report, the FBI’s IT was severely outdated by the time the Trilogy project was initiated in 2001.

In the current CIO’s vision, the counterterrorism mission of the FBI is supported by three functions that must revolve around IT:
1) data collection, 2) data analysis and investigation, and
3) dissemination of information. IT, as the current CIO sees it, is the enabler that will allow agents and analysts to perform these three functions, and Trilogy specifically was intended to provide a solid framework in performing those functions. However, because the VCF is incomplete, that framework has only been enhanced in two ways.

First, the IPC portion of Trilogy has placed more modern equipment on the desk of every agent and analyst, when previously agents and analysts had to share computers that in some cases were
8 to 12 years old. The IPC modernization process allows agents to access FBI systems immediately. Additionally, the new computers include a standardized set of applications for e-mail, word processing,
databases, and spreadsheets. As a result, the ability for information sharing among agents, analysts, and other support personnel is enhanced because all employees are working within the same formats and structures.

Second, the connectivity of the Trilogy network provides electronic communications capability and access to investigative and administrative information. While the FBI’s previous system did not allow for data to be transferred among agents and analysts, the new system allows for sharing large files, such as photographs. Additionally, the modernized system allows for better encryption of the data transmitted among FBI staff, as well as better storage of the information.

The FBI believes that the system resulting from the FICMS effort will replace the limited and antiquated ACS system. The ACS data management system has not been fully utilized by agents in performing their casework. Consequently, the data contained in the system does not necessarily represent all of the information for a specific investigative case file. This requires agents to search more than one database in an effort to obtain all the documents relating to a case. Additionally, the functionality and operation of the ACS is not user-friendly and requires the input of several pieces of information to perform basic tasks. The version of the VCF due by December 31, 2004, (the Initial Operational Capability or IOC) is intended to demonstrate that case documents can be approved electronically and uploaded into the ACS. However, this version of the VCF does not provide the case management and information-sharing system envisioned for Trilogy.

In addition to the IOC, the FBI has developed an interim step to make the ACS more user-friendly by making the system accessible through the FBI’s intranet. The web-based ACS reduces the number of steps needed to operate the ACS from 12 to 3 and offers other features to make the ACS easier to use. However, the FBI’s need for a fully functional case management system to replace the technologically obsolete ACS remains urgent.

Improvements in IT Management

Despite the problems in completing Trilogy, it is important to note that the FBI is making progress to improve its IT management.
After appointing a new CIO in May 2004, the FBI reorganized its IT resources in July 2004. The FBI established the Office of the CIO to centrally manage all IT responsibilities, activities, policies, and employees across the FBI. As mentioned earlier, one of the problems cited in the OIG’s audit of the FBI’s management of IT investments was that all of the FBI divisions that had IT investments were not under a single authority and, as a result, had disparate management processes and procedures. With the FBI’s new IT organization, all IT projects now fall under the Office of the CIO. (Appendix 5 shows the organizational chart for the new Office of the CIO.) The CIO has the responsibility for the FBI’s overall IT efforts, including developing the FBI’s IT strategic plan and operating budget, developing and maintaining the FBI’s technology assets, and providing technical direction for the re-engineering of FBI business processes.

The Office of the CIO has also developed an FBI-wide Life Cycle Management Directive to guide FBI personnel on the technical management and engineering practices used to plan, acquire, operate, maintain, and replace IT systems and services. The directive provides detailed guidance to each FBI Program/Project Manager charged with managing programs or projects throughout the life cycle from inception to deactivation. The processes and procedures, if fully implemented, should help prevent the delays and problems that occurred during the Trilogy project.

The FBI’s 2004–2009 Strategic Plan includes the objective to "ensure that all current and future information technology plans work towards a harmonized system." This objective would be met through the creation of an Enterprise Architecture. The FBI is in the process of creating an Enterprise Architecture by September 2005 through its IT Portfolio Management Program.

The FBI’s IT Portfolio Management Program is a phased process where the documentation of the FBI’s enterprise-wide IT portfolio is established or, in lay terms, a listing that contains all of the FBI’s current IT systems will be created, including the technical
documentation of those systems. The first phase in creating the listing, completed in February 2004, focused on a pilot performance assessment of Information Resources Division applications. The second phase of the program, the infrastructure portfolio assessment, was initiated in March 2004. Once the data collection for this phase is completed, the next steps include workshops to begin assessing the division’s current infrastructure. Completion of the portfolio assessment was targeted for late 2004, and the FBI anticipates that the recommendations from the completed portfolio will be included in the development of the FY 2007 budget.20

Federal Intelligence Information Reports Dissemination System

In addition to the VCF IOC, the FBI’s CIO pointed to the recent development of the Federal Intelligence Information Reports Dissemination System (FIDS) as an example of how the FBI’s reorganized IT management structure can successfully implement a significant project, although FIDS does not represent an application as complex as the VCF or involve case management functions. FIDS is a web-based system for creating the FBI’s Intelligence Information Reports (IIR), a primary intelligence product intended for dissemination throughout the U.S. intelligence community. Through FIDS, the drafting of IIRs is automated and standardized, and they can be approved electronically after supervisory review. Because of the computer language used, FIDs is interoperable with other intelligence agencies’ systems. Further, a variety of attachments are possible including photographs, video, and audio. The CIO said that the system complies with the information-sharing requirements of Executive Order 13356.

According to the CIO, the FBI developed FIDS in six months for $350,000 by adapting an existing secure system used for Foreign Investigative Surveillance Act requests. The FBI identified, prioritized, and froze the requirements prior to developing the system. The CIO said that FIDs results in a standardized FBI-wide format for IIRs and allows the IIRs to be prepared, approved, and disseminated much more rapidly. This reporting function, however, does not represent the significant systems that will be required for a case management system, and FIDS was only used to demonstrate a recently developed project that includes some of the functionality a case management system would include, such as supervisory reviews and information dissemination.

Conclusion

The FBI has made progress in modernizing its IT infrastructure by installing modern computers and providing a secure network. However, the FBI has not been able to achieve one of the most critical requirements of the Trilogy project, development and deployment of
the investigative case management capabilities of the VCF. The FBI does not know how long it will take to achieve this fundamental goal or what this process will ultimately cost. After more than three years of attempted development of the VCF at an estimated cost of $170 million, FBI employees are still using the outdated and inadequate ACS system. While during this process the FBI likely has advanced its understanding of the requirements needed for a future case management system, the $170 million obligated to date on development of the VCF has not fulfilled the FBI’s goal of producing a usable case management system.

We found a number of reasons for the FBI’s difficulties in completing the Trilogy project successfully and on time. One major reason for the delays and cost growth in the overall project was a lack of specific design requirements for each of the project components. Only with specific requirements in place can the methods, costs, and timeframes for implementation be determined. Because the FBI did not establish at the outset what was needed for the project and establish how those needs should be implemented, the cost of the project ballooned and the implementation schedule slipped. The effect of the lack of specific design requirements was worsened later by necessary modifications.

The Trilogy project, having been initiated prior to September 11, focused on modernizing the FBI’s infrastructure to make ACS a
user-friendly system. But once the FBI acknowledged that ACS was too archaic a system, the need for the VCF became evident. Such changes could have been more easily managed without severe schedule and cost implications had the technical and performance requirements been established and the project managed more effectively.

The FBI’s decision to use a cost-plus-award-fee contract to develop Trilogy placed it at a significant disadvantage because the contract did not establish firm milestones or prescribe penalties for a contractor that missed deadlines or delivered an unacceptable product. According to the FBI, it used this type of contract because it wanted to move forward on the Trilogy project expeditiously. Instead, however, this process resulted in delays and, with respect to the VCF, ultimately did not result in a fully functioning product. In addition to an unsuitable contracting mechanism and the lack of firm design requirements, the FBI also lacked sound IT management processes, continuity of project management and oversight, and project
integration and engineering skills. Further, FBI management did not act in a timely manner on a number of critical internal and external reports that demonstrated significant project risks. Had well-established investment management practices been in place earlier, the FBI’s IT needs may have been defined better, possibly avoiding the misstep of initially pursuing merely a modification of the ACS system. In addition, a professional project integrator might have recognized more quickly the scheduling setbacks for both the infrastructure and user application components.

FBI management did not exercise adequate control over the Trilogy project and its evolution in the early years of the project.
It now appears that with the new organizational structure and authority given to the CIO in July 2004, project management is receiving the attention that was needed throughout the Trilogy project.

Still, the original UAC portion of the Trilogy project remains incomplete despite the FBI’s allocation of $170 million for the VCF. Moreover, the cost and schedule for full implementation of the successor to the VCF through the interagency FICMS effort, is unknown because: 1) the FBI has not fully established the requirements for the case management system, and 2) no contract will be in place to develop the framework for this new case management system before the third quarter of FY 2005.

In sum, we believe the FBI’s ability to perform its important functions effectively, including counterterrorism, counterintelligence, and criminal law enforcement, will be significantly affected by its ability to implement a modern case management system.

Recommendations

We recommend that the FBI:

Replace the obsolete ACS system as quickly and as cost-effectively as feasible.

Freeze the critical design requirements for the case management system before initiating a new contract and ensure that contractor fully understands the requirements and has the capability to meet them.

Incorporate development efforts for the VCF into the development of the requirements for any successor case management system.

Develop policies and procedures to ensure that future contracts for Trilogy-related projects include defined requirements, progress milestones, and penalties for deviations from the baselines.

Establish management controls and accountability to ensure that baselines for the remainder of the current user applications contract and any successor Trilogy-related contracts are met.

Apply ITIM processes to all Trilogy-related, or successor, projects.

Monitor the Enterprise Architecture being developed to ensure timely completion as scheduled.

Footnotes

An award fee is a financial incentive provided to a contractor, based on the contractor's performance, as a form of motivation to meet directed baselines.

In commenting on a draft of this report, the FBI stated that the five primary applications were selected because they were the applications used most frequently by agents in support of investigative activities and that this was a reasonable set of applications to pursure given the original approach of "webifying" the ACS.

However, while the IIIA application's functionality was to be captured within the VCF, the migration of the IIIA data may not be part of the VCF because, according to one VCF project manager, agents utilized the system inconsistently and the data within the application may not be complete. The FBI told us it is reviewing the IIIA data to determine if it will be captured in the ACS data migration to the VCF.

The RIA component consists of current data collected within a file whenever an agent performs investigative work on a lead or case. The data within the RIA would be entered into the VCF within established data fields. An enhancement to the VCF would allow for the data to be compared to other leads or case files.

In commenting on a draft of this report, the FBI stated that Track Two is in essence the FICMS effort that is expected to eventually result in a fully functional investigative case management system.

The FBI's CIO stated that the FBI's involvement in FICMS stems from both the Office of Management and Budget and Executive Order 13356 entitled Strengthening the Sharing of Terrorism Information to Protect Americans, signed on August 27, 2004 (see Appendix 4). The Executive Order provides guidance to organizations that possess or acquire terrorism information on issues relating to the sharing of such information among agencies. Part of the guidance is to establish an interoperable terrorism information sharing environment to facilitate automated sharing of terrorism information among appropriate agencies.

In commenting on a draft of this report, however, the FBI disputed the project manager's explanation, saying that security requirements were initially identified within the base contract. Instead, the FBI cited the addition of a new e-mail function as the primary reason for the schedule delays along with the implementation of a product assurance program.

An award fee is a financial incentive provided to a contractor, based on the contractor's performance, as a form of motivation to meet directed baselines.

In commenting on a draft of this report, the FBI agreed that it was unfamiliar with SAIC's scheduling tool but said it was still able to validate and analyze the schedule and use that information in managing the project.

In commenting on a draft of this report, the FBI cited the VCF IOC as demonstrating the advances made in its program management processes to achieve cost and schedule baselines. For example, the FBI listed the following improvements: assigning dedicated individuals in key project roles; clearly defining objectives, requirements, and constraints; planning; using control gates at major milestones to release funding and keep the project focused; configuration management; co-locating government and contractor staff; collaborating with other FBI divisions; and providing project oversight at all levels within and also outside the FBI. Because the VCF IOC was incomplete at the time of our audit, the OIG could not evaluate the claims about recent process improvements cited by the FBI.