Additional Materials:

Contact:

We are reviewing the processes the Department of Defense (DOD) is using to conduct its Mobility Capabilities Study (MCS). The MCS is to address changes in DOD's transportation force structure and mobility requirements due to changes in threats and certain national security and military strategies. The study results may underpin decisions on future strategic airlift, aerial refueling aircraft, and sealift procurements. The study relies on the use of various models and data inputs to develop and evaluate transportation alternatives, including variations in alternative transportation modes (air, land, sea) and sources (military, civilian, foreign), as well as factors that affect transportation mode and source decisions. The Senate Armed Services Committee directed us to monitor the conduct of the MCS and report on the adequacy and completeness of the report no later than 30 days after DOD completes the study. DOD plans to issue the MCS report during 2005. This letter is intended to bring to the Secretary of Defense's attention preliminary observations on certain aspects of the MCS methodology to permit you to ensure the credibility of this and future studies. In our letter, we address the adequacy of the department's verification, validation, and accreditation (VV&A) of the models and simulations being used to conduct the MCS--that is, the process the MCS team is using to identify the models' capabilities, limitations, and performance relative to the real world events they simulate. We will continue to monitor the MCS and will report on the adequacy and completeness of the methodology after DOD issues its report.

We are unable to assess the adequacy of the process DOD used to verify, validate, and accredit the models used to conduct the MCS. Although officials in the Office of Program Analysis and Evaluation stated that they have performed an equivalent VV&A process for the models used in the MCS, there is little documentation available to describe the equivalent process that was used. An adequate evaluation of this self-described equivalent process cannot be conducted due to this absence of documentation, which is compounded because DOD currently does not plan to disclose how it conducted its equivalent VV&A process in its MCS report. This could negatively impact the credibility of the MCS report. DOD guidance requires that models and data go through a VV&A process, but officials in the Office of Program Analysis and Evaluation believe that this guidance is not relevant for models that have been used for many years, called legacy models, because, in their view, the models and data have already undergone an equivalent VV&A process consisting of actual use, although the guidance does not identify actual use as an appropriate equivalent process. Moreover, DOD was conducting VV&A on one legacy model being used in the MCS, raising questions about the need for such actions given the department's statements that it is unnecessary.

Recommendations for Executive Action

Status: Closed - Not Implemented

Comments: DOD states that these models have been used for years and their strengths and weaknesses are known and adjusted for. Officials don't see the need to do a VV&A on older models.

Recommendation: The Secretary of Defense should direct the Director, Office of Program Analysis and Evaluation, to develop documentation that describes the equivalent VV&A process used to verify and validate the mobility models and baseline data used to conduct the MCS prior to publishing any portion of the study results.

Agency Affected: Department of Defense

Status: Closed - Not Implemented

Comments: DOD did not disclose the equivalent process in the MCS report.

Recommendation: The Secretary of Defense should direct the Director, Office of Program Analysis and Evaluation, to disclose in the published MCS report the equivalent VV&A process used on the models and baseline data.

Agency Affected: Department of Defense

Status: Closed - Not Implemented

Comments: DOD has no plans for reviewing the VV&A guidance.

Recommendation: The Secretary of Defense should direct the Director, Office of Program Analysis and Evaluation, to work with the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics to evaluate the current DOD VV&A guidance to determine its relevance for use with legacy models and to change the guidance if appropriate.