Executive Summary

This report summarizes the results of an evaluation of the Travel Model Improvement Program (TMIP) Peer Review Program. The purpose of the evaluation was to understand the value to transportation planning host agencies of a peer review of their travel demand model. The study involved interviewing modeling staff from eight metropolitan planning agencies that have hosted a TMIP-sponsored peer review of their travel demand model.

Major findings are detailed below, grouped into four categories: motivations for conducting a peer review, planning the peer review, implementing technical recommendations, and overall satisfaction with the process. Following the findings, recommendations for enhancing peer reviews and the peer review process are presented.

AGENCY MOTIVATIONS FOR CONDUCTING A PEER REVIEW

One of the major advantages of peer reviews is that each host agency can frame the meeting to address its unique modeling needs. Nonetheless, there were some common reasons for holding peer reviews:

Assess the model with respect to the state of the practice.

Obtain comments and advice on the model's compliance with Federal requirements.

Identify and build support for modeling activities among agency senior management and policy-makers.

Help determine priorities for model enhancements.

Grow the skills of the agency modeling staff.

PLANNING THE PEER REVIEW

Interviewees agreed that planning their peer review was very time consuming. However, they also agreed that good planning is a critical factor in a successful peer review. Planning tasks that interviewees found particularly important were:

Choosing the best panelists for the peer review.

Deciding the length and number of peer review meetings.

Providing peer panelists with documentation of technical details of the model and specific issues to be addressed in the peer review.

Consulting with senior management and policy-makers to gain their support and input for the peer review.

IMPLEMENTATION OF RECOMMENDATIONS

The most common technical issues addressed in peer reviews were:

Model platform and land use integration

Activity-based versus four-step models

General model details and data needs

Freight modeling

Mode choice

Validation, calibration, and feedback

Agencies implemented approximately 70 percent of peer panelists' recommendations. The major reasons for not implementing recommendations were:

Lack of resources.

State of the practice. In some cases, the technology and applications necessary for implementing a recommendation were not sufficiently mature for agencies that did not want to incur the risk and dedicate the additional resources necessary to develop and use relatively untested technology or applications.

Timing and priorities. Some agencies had higher priorities-such as preparing the transportation improvement plan-than implementing the recommendations right away.

Agency opinion. In some cases, the agency decided not to implement the recommendation because, after learning more about it, the agency decided that the recommendation was not appropriate for its circumstances.

OVERALL SATISFACTION

Interviewees were very satisfied with their peer review and the peer review process. They felt that panelists were highly skilled and that the recommendations were appropriate and helpful. The post-meeting report was helpful in documenting these recommendations. Interviewees said that the peer review helped build their staff's modeling skills and help improve their travel demand model.

RECOMMENDATIONS FOR IMPROVEMENTS TO THE TMIP PEER REVIEW PROGRAM

The following recommendations are drawn from the interviews as well as from observations of the Volpe Center team.

Develop outreach materials for senior managers to inform them of the purpose and potential benefits of a peer review.

Provide more guidance and technical assistance to agencies for hosting a peer review.

Conduct pre- and post-panel conference calls with panelists.

Identify and inform the panel of agency constraints prior to the meeting.

Provide more technical details in the post-panel report to help agency staff understand the reasoning behind the recommendations.

Create a database that includes agencies and the issues discussed in their peer review for other agencies to use in planning peer review meetings.

Provide agencies with suggestions and advice on how to prepare an RFP or interview consultants.

Continue to conduct periodic evaluations to monitor the effectiveness of the peer review program.

Continue to develop and disseminate materials on "state of the art" and "state of the practice" in travel demand modeling.

Continue to work with modeling agencies to build peer networks.

Abbreviations

BMC

Baltimore Metropolitan Council

COMPASS

Community Planning Association of Southwest Idaho

DRCOG

Denver Regional Council of Governments

EWGCOG

East-West Gateway Council of Governments

NJTPA

North Jersey Transportation Planning Authority

SANDAG

San Diego Association of Governments

SEMCOG

Southeast Michigan Council of Governments

SCAG

Southern California Association of Governments

TMIP

Travel Model Improvement Program

I. Introduction

This report summarizes the results of an evaluation of the Travel Model Improvement Program (TMIP) Peer Review Program. The purpose of the evaluation was to understand the value to transportation planning host agencies of a peer review of their travel demand model. This report presents strengths and weaknesses of peer review meetings and the peer review program as described by modeling staff from agencies that have hosted a peer review. It also provides recommendations for improvements to the TMIP Peer Review Program and to aspects of the overall TMIP Program that could help grow the educational impact of peer reviews.

TMIP is a partnership between the Federal Highway Administration (FHWA) and the Federal Transit Administration (FTA). The program's goals are to:

Help planning agencies build their institutional capacity to develop and deliver travel-related information to support transportation and planning decisions;

Develop and improve analytical methods that respond to the needs of the planning and environmental decision-making processes; and

Develop mechanisms to ensure the quality of technical analysis used to support decision-making and to meet local, state, and Federal program requirements.

Improved modeling techniques allow planners to generate better, more robust transportation forecasts that decision-makers can use when making important decisions about transportation services under various scenarios changes in population, economic growth, and land use.

TMIP's peer review component gives transportation planning agencies the opportunity to have their model reviewed by modeling experts from around the country. These experts make recommendations on how to proceed with model enhancements. Peer reviews are particularly valuable because each host agency customizes its peer review to meet its unique modeling concerns.

This study examined peer reviews conducted by eight metropolitan planning organizations (MPO) across the country. Section II of this report describes the methodology used in the study. Section III presents findings drawn from major themes in the interviews. Section IV presents recommendations for peer review program enhancements, followed by the conclusion in Section V.

II. Methodology

This evaluation was conducted by the U.S Department of Transportation (DOT), Research and Innovative Technology Administration (RITA), Volpe National Transportation Systems Center (Volpe Center) at the request of the FHWA Office of Planning, Environment, and Realty. It consisted of eight semi-structured telephone interviews with travel demand modeling staff from eight transportation agencies that have hosted peer reviews. Five of these eight agencies conducted their peer review after 2005 when the most recent peer review evaluation was conducted. These five agencies, therefore, have not been evaluated previously.

FHWA chose three additional agencies to provide a more balanced perspective of the program. Each of these additional agencies conducted its peer review at least 5 years ago, giving them substantially more time to address peer review recommendations than the five agencies that conducted peer reviews since 2005. This evaluation focused more on agencies that are in the process of developing their model rather than refining an existing model. Four agencies studied have conducted more than one TMIP-sponsored peer review. This evaluation focused on the most recent peer review.

III. Findings

This section lays out the common themes expressed by interviewees. They are divided into four categories: motivations for holding a peer review, planning the peer review, implementation of technical recommendations, and overall satisfaction.

A.MOTIVATIONS FOR HOLDING A PEER REVIEW

Agencies were looking to the peer review panelists for expert advice on the best approaches to improve their model's capability in high-priority areas. Common reasons for requesting a peer review included:

Assess the model with respect to the state of the practice.
Several interviewees said that one of the reasons for their peer review was to get an "independent eye" to review their model and comment on it with respect to the state of the practice. COMPASS, for example, requested that the panel provide general commentary and advice on its model. NJTPA, which was in the process of making major enhancements to its existing model, wanted "to make sure it was on the right track; be sure we were using the best modeling techniques available." At SEMCOG, management wanted the peer review "to be sure that the modeling process was robust and have people in the region feel comfortable with the process."

Provide expert opinion to help agencies respond to controversies about the model, model components, or modeling process.
Output from a travel demand model can have a substantial effect on transportation policy and, subsequently, air quality, land use, and economic development planning. By running different scenarios, the projected transportation conditions and needs are used to inform short- and long-term transportation planning.

Occasionally, stakeholders-usually environmental or neighborhood-based groups-object to local plans based, at least in part, on model output. Some stakeholders question the model itself, contending that the data, assumptions, or algorithm are not consistent with the state of the practice. Many agencies requested a peer review in part to address the concerns of these stakeholders. When determining the charge to the peer review panel, these agencies specifically requested comments on controversial model elements either to demonstrate that the model is reasonable or to identify model deficiencies and suggest improvements.

For example, an environmental group in Baltimore questioned the output of the travel demand and air quality models in forecasting the impact of the transportation system on air quality. For BMC, the peer review process helped to provide legitimacy to the model outputs based on an assessment by objective, outside modeling experts. In San Diego, SANDAG requested a peer review partly to respond to criticism about its model by soliciting panelists' comments addressing mode share for future transit systems.

Receive technical assistance on particular model components.
Most agencies made specific requests that the panel review one or several technical model components. For example, SANDAG requested guidance on the best forecasting methodologies for evaluating new transit services and smart growth and land use policies. EWGCOG asked for comments on its volume-delay function and on detailed aspects of the mode choice model. Several agencies requested advice on model calibration and validation.

Identify and build support for funding modeling activities.
Some agencies wanted to use the peer review comments and recommendations to demonstrate the agency's modeling needs as assessed by objective outside experts. They hoped that this would encourage decision-makers to provide sufficient resources to maintain and enhance modeling activities. For example, DRCOG, which conducted a deliberate and careful visioning process for its new approach to modeling, said that "the weight of the [peer panel] experts' names" helped build political support for its new modeling effort.

Receive comments and advice on the model's compliance with Federal requirements.
SANDAG and BMC each specifically requested comments and advice on their model's sufficiency with respect to FTA's New Starts requirements. While these were the only agencies that specifically requested comments on New Starts funding criteria, most of the peer reviews included some discussion of New Starts modeling requirements. Even agencies that were not planning to apply for New Starts funding wanted a model that would meet FTA criteria in case the agency decides to apply for New Starts money in the future. Although complying with New Starts requirements is important to many transportation agencies, in order to avoid debating modeling and modeling requirements with FTA, TMIP does not approve peer review requests in which the agency's only goal is to receive feedback on its model with respect to FTA New Start requirements.

In addition to New Starts requirements, travel demand models must meet Federal requirements for certification of the region's planning process. Because transportation modeling is highly technical and every model is unique, there are no specific Federal modeling requirements. When evaluating the planning process, Federal agencies generally consider a model peer review as evidence that the model and modeling process is consistent with the state of the practice or that the agency is working to improve its model. One interviewee noted that the peer review "provided good input to the certification process."

Help determine priorities for model enhancements.
Every agency studied asked the panel for recommendations on short- and long-term modeling priorities. On this request, agencies generally fell into two categories: those that had a good idea where they were going but wanted reassurance that they had chosen the most efficient way to get there; and those that were at a standstill and did not know which direction to take for model enhancements.

Of the former, some of the agencies were looking to the panelists' recommendations in part to validate their plans for model improvements, hoping to reassure politicians and senior managers of the legitimacy and importance of the planned enhancements. This, they hoped, would convince decision-makers to provide sufficient funding for implementing model enhancements. Agencies in the latter category-COMPASS, for example-wanted a general sense of the state of their model relative to other travel demand models and advice on how to allocate their resources to meet their agencies' modeling objectives.

Receive comments and advice for modeling prospective policy alternatives.
Many metropolitan areas are considering major changes to their transportation network and its operations. Some changes are pro-active policy initiatives-such as congestion pricing or high-occupancy vehicle lanes-to control congestion and generate revenue for today and the future. Other proposed policy alternatives are intended to address problems largely outside the control of planning agencies such as the merging of neighboring metropolitan areas into a single urban conglomerate.

Modeling many of the major new policy initiatives can be extremely complicated and risky since these techniques are still in their infancy. For agencies that do not intend to develop a new model to accommodate these policy initiatives-most of those included in this study-peer panelists have suggested less complicated techniques such as post processing or add-ons.

Grow the skills of the agency modeling staff.
Many of the Nation's foremost experts in travel demand modeling have served as peer reviewers. This has given agency modeling staff an important opportunity to increase their modeling expertise-formally through the peer review meeting, discussions, and peer panelists recommendations and informally by building relationships with nationwide experts and expanding their professional network.

B.PLANNING THE PEER REVIEW

Planning a peer review requires significant effort on the part of the host agency. Interviewees made several observations about the process of planning the peer review:

Involving senior management and policy-makers in planning the peer review strengthens the process and improves outcomes.
Some interviewees said that it was important to communicate to senior managers and policy-makers the strengths and weaknesses of the agency's current model, to describe the implications of these characteristics, and to demonstrate how a peer review can help improve its model and modeling process. Senior managers and policy-makers should not be expected to weigh in on specific model elements or on the panel's technical recommendations. However, by helping senior managers and policy-makers understand the value and limitations of the peer review, they can make better-informed decisions about allocating resources to improve their model.

Planning a peer review requires substantial time and effort but is critical to a successful peer review.
Agencies are responsible for developing the charge to the panel, setting the meeting schedule and agenda, selecting panel members, providing background material, developing presentations to be given at the peer review, and planning meeting logistics. One interviewee noted that the host agency "needs to be clear about the agenda and do advance preparation... to help clarify key points of confusion and determine the magic number of participants and narrow the panel discussion." Interviewees agree that this required a lot of effort, but that doing this well is necessary to get the most out of their peer review. The following are detailed findings related to these planning activities:

Choosing panelists with expertise in topics or circumstances important to the host agency. Interviewees said that, when choosing panelists, they wanted at least one person with experience modeling for agencies in areas similar in size and geography of the host agency. Panelists representing a broad range of expertise were also essential to a successful meeting. The typical panel included representatives from consultants, academia, MPOs, and other government agencies. This range of experience was important to provide a broad perspective on the intricacies of travel demand modeling.

However, recruiting panelists is more than locating modelers with a desired expertise. Once panelists have been identified, each has to be available to participate; and the host agency must choose a date for the peer review meeting that is convenient for all panelists. Though the effort was complicated, interviewees agreed that getting the right panelists is very important.

Deciding the length and number of peer review meetings. Some agencies chose to split the process into two or three meetings. After the first meeting, the agencies had time to act on the recommendations. The second and subsequent meetings consisted of discussion of these recommendations and the agency's progress in addressing them. It also provided agencies the opportunity to ask the panel to address new modeling issues that have come up since the first meeting.

BMC held two peer review sessions. The first meeting in September 2004 focused on a review of the status of its model improvement process. Panelists provided guidance on near- and long-term model development issues. The second meeting, held in February of 2005, involved BMC presentations of progress made from the first meeting and included discussion on developing its work plan to address the panel's recommendations.

DRCOG hosted two meetings while it was in the process of developing a completely new model. It used the first meeting to discuss the agency's visioning process with the panelists. DRCOG wanted to use the panel's expertise to educate itself on advances in forecasting models beyond the recommendations it had received from its consultant. Its second panel, held six months later, focused on the preliminary suggestions for the new model blueprint.

Providing panelists with technical details of the model well before the peer review meeting. The host agency is expected to provide detailed documentation of its model and modeling process several weeks before the peer review so that peer panelists and other attendees arrive at the meeting with a good understanding of the model's technical details and the specific problems the peer review will address. Interviewees said that writing this background material was very time consuming. However, they felt that this documentation was very important because it eliminated the need to use meeting time to explain model intricacies.

C.IMPLEMENTATION OF RECOMMENDATIONS

Table 2 lists the most common topics for recommendation and their status of implementation. It shows that host agencies implemented approximately 70 percent of the panels' recommendations. In addition, agencies either have implemented or are considering implementing 23 percent of the recommendations. Seven percent were not addressed.

Agencies were most likely to implement recommendations related to basic model elements such as modeling boundaries and network revisions. Other frequently implemented recommendations were related to mode choice, validation, and calibration. Altogether, agencies implemented over 80 percent of the recommendations that address basic model elements.

Table 2: Implementation of TMIP Panel Recommendations

Agency/ Recommendation Topics

Model Platform, Land Use Integration

Activity-Based Modeling

General Model Details/Data Needs

Freight Component

Mode Choice/ New Starts

Validation/ Calibration/ Feedback

BMC

√

+

√

-

√

+

√

o

COMPASS

√

+

√

o

√

-

√

o

DRCOG

√

o

√

+

√

+

√

-

√

+

√

+

EWGCOG

√

o

√

+

√

o

√

+

√

+

NJTPA

√

+

√

o

√

o

√

+

√

+

SANDAG

√

o

√

o

√

+

√

+

√

+

SCAG

√

+

√

+

√

+

√

+

SEMCOG

√

o

√

o

√

+

√

o

√

+

√

+

Key:

+

Implemented

o

Considered/partially implemented

-

Not implemented

√

Focus of discussion

Several recommendations were partially implemented or delayed because of:

Lack of resources: Some interviewees indicated that the agency did not necessarily disagree with the recommendation; rather, it did not have the staff or funding to hire a consultant for the task. These types of recommendations included additional data collection, land use integration, and migration to activity-based models.

State of the practice: In some cases, agencies did not implement recommendations because the state of the practice for implementation is not well developed, introducing expenses and risks that the agency felt were unacceptable.

Timing: Some agencies delayed implementing recommendations for major model revisions because they were occupied with other important tasks such as developing long-range transportation plans. One agency delayed acting on recommendations until after its Regional Transportation Plan was complete.

Priorities: Upon hearing the peer review panel's recommendations on particular model aspects, some agencies decided that implementing some of the recommendations was simply not a high priority for their agency. This was most common with recommendations related to freight because of resource constraints, the proprietary nature of information related to the freight industry, or lack of support from state departments of transportation.

Agency opinion: In a few cases, agencies felt that the panel's recommendations were simply not appropriate for their model. One agency indicated that the panelists recommended modifying network travel times without having a clear understanding of the agency's rural roadway system where travel times were not of as much concern. Another agency felt that the recommendation to refine market segmentation based on auto ownership was unnecessary.

None of the interviewees said they could not implement a particular recommendation because agency decision-makers disagreed with it. This is to be expected since most agencies vetted issues with management before deciding the charge to the peer review panel.

The list below presents details on the most common technical topics addressed in peer reviews:

Model Platform/Land Use Integration: The larger MPOs discussed with panelists the possibility of integrating land use with the travel demand model to better capture development patterns. In some cases, the panels encouraged agencies to work toward this goal. However, interviewees agreed that the state of the practice is evolving; and agencies must be aware of the resources and risks involved.

Activity-Based versus four-Step Modeling: A few panels recommended that agencies begin to develop a tour- or activity-based model but warned of the costs, expertise, and risks involved. One agency included in this study has begun tour- or activity-based modeling. Two other agencies have plans to implement tour-based modeling with future model updates. A fourth is staying with its four-step model because it felt that it was too far along in the process of developing its four-step model to make such a dramatic change. This agency will consider tour- or activity-based modeling for its next set of major model improvements.

General Model Details and Data Needs: Each of the peer reviews studied for this evaluation included discussion of general improvements to basic model components such as networks, traffic analysis zones, and model boundaries. Many discussed the need for additional, current, and accurate data on elements such as travel speeds, traveler behavior, regional employment forecasts, external trips, transit use, and commercial vehicle travel. In most cases, these recommendations were implemented or are included in the agency's Unified Planning Work Program for future implementation. A few interviewees said financial constraints prevented them from moving forward on data collection, but they planned to collect data as their budget allows. Panelists strongly encouraged organizations to share data if possible.

Freight Model: Panels consistently recommended that agencies work to understand the effect of commercial vehicles on the transportation network. Some interviewees said their agency has not yet developed a freight component because it does not have the necessary funding or management support. Panels recommended working with the state department of transportation, although in some cases, states do not have the necessary data or staff to support a freight model.

SCAG, for example, indicated that the panel "did a very good job of identifying future areas of improvement for our heavy-duty truck model...and the improvements listed in the peer review report are a basis for model improvements we are seeking to implement through this effort." However, this interviewee also noted that "obtaining quality freight and commodity flow data at a detailed geographic level and quality survey data remains a major challenge."

Mode Choice: Generally, interviewees said that the panelists made recommendations to improve their mode choice model, especially by collecting data to validate it. Agencies from large metropolitan areas were consistently concerned with meeting the modeling requirements for the FTA New Starts Program. A few interviewees noted that they wanted advice on how to best integrate bus-rapid-transit into their model. In a few locations, panelists recommended including non-motorized transportation. Every agency felt these were worthy recommendations although financial constraints have hampered their implementation in some areas.

Validation, Calibration, and Feedback: Every agency asked its peer review panel to address model calibration and validation. Panelists recommended calibrating and validating the model to customized targets rather than assignment standards with feedback. Most agencies implemented these recommendations. One agency was planning to incorporate the work into the next fiscal year's work program. Another agency felt that the panel did not understand its roadway system so it did not plan to make all the recommended changes.

Interviewees agreed that the technical recommendations were helpful and resulted in significant improvement to their models. However, one agency acknowledged that the comments and advice provided by the panel could be a "double-edged sword." With so many possible model enhancements, panels occasionally presented an overwhelming list of options that the agency could not even consider given staffing constraints and the maturity of its current model.

D.OVERALL SATISFACTION

The evaluation interviews asked about overall satisfaction with the peer review process. Interviewees were generally very satisfied with the process and made the following key points:

Most of the technical recommendations were helpful and resulted in significant model improvements.

When asked to rate the suitability of the TMIP process in achieving the agency's major outcomes on a scale of 1 to 5 (with 5 being the best rating), interviewees gave the process an average rating of 4.25. In the discussions, they noted specific examples of how implementing the panel's recommendations improved its model:

Provided guidance on improved representation of system performance measures.

Recommended including trip chaining, which improved model results.

Helped prepare the agency for discussing New Starts grants with FTA.

Provided guidance on obtaining new land use integration software that the agency has successfully used.

One interviewee cited an example of a recommendation that was not helpful. The panel's comments on the rail constant in mode choice were very different from what FTA wanted for New Starts funding. As a caveat, however, the interviewee noted that the model as presented in the peer review was too far off the targets for the panel to give effective, informed recommendations.

The peer review format was very effective.

When asked to rate the effectiveness of the meeting format on a scale of 1 to 5 (with 5 being the best rating), interviewees gave the process an average rating of 4.44. Because each agency created its own agenda, staff were able to develop the format that worked best for them. All interviews agreed that this arrangement worked well.

Panelists were very professional and provided valuable technical assessments.

When asked to rate the suitability of the panelists on a scale of 1 to 5 (with 5 being the best rating), interviewees gave the panelists an average rating of 4.69. Interviewees said that the panelists' honest and objective judgment was an important part of the peer review. The panelists' breadth of experience in topics including land use, transportation policy, and technical modeling components gave agencies a broad perspective on the potential for their models.

One interviewee noted that the high-caliber reputation of the panelists was helpful in encouraging its consultants to participate in the peer review. Before the agency began preparing for the peer review, the consultant discontinued work on the agency's model until receiving additional funding. The panel's potential influence on the consultant's professional reputation gave the agency the leverage it needed for the consultant to complete the work necessary to present a respectable model to the peer experts.

One interviewee noted that one panelist was somewhat distracting to the process because of his narrow focus on a particular issue that was not part of the questions asked of the panel. This example was an exception to the overall positive feedback on the panelists' professionalism and technical expertise.

The peer review report was a valuable resource for implementing recommendations.

Interviewees said that their agencies refer back to the report frequently for guidance on how to move forward with changes to the model. Every interviewee said that a timely and accurate report on the meeting was extremely important, particularly because it contained details of the panel's recommendations. One interviewee said he was "glad to have an independent party write the report to make sure that the outside ideas were represented. It could have been tempting for our agency to represent what it wanted to hear rather than what actually happened." Interviewees agreed that receiving the report within about 2 months after the meeting was important so that agencies could quickly move forward in addressing recommendations.

Participation in the review process helped agency staff and peer panelists to expand their professional network.

Peer reviews brought together a variety of planning and modeling professionals. Interviewees said that this helped to build relationships among the various peer review attendees. The opportunity for networking was especially beneficial for agencies located outside of major metropolitan areas where they are not likely to interact with other modeling professionals locally. Some interviewees indicated that occasionally host agency staff continued to follow up with members of the panel after the meeting for technical advice or guidance on recommendations. They felt that the peer review was a successful technique for creating sustained relationships among planners and modelers.

IV. Recommendations

The following recommendations are drawn from the interviews as well as from observations of the Volpe Center team.

Develop outreach materials for senior managers to inform them of the purpose and potential benefits of a peer review.

Interviewees felt that it was extremely important that senior managers and decision-makers understand and approve of the peer review process prior to the meeting. Managers should appreciate the potential benefits of a peer review but also have realistic expectations for the outcomes of the peer review. TMIP staff should consider developing outreach materials stressing the following points:

Recommendations are not requirements but rather suggestions to improve the model.

Peer reviews are looked upon favorably by Federal agencies when considering planning certification.

Panelists are modeling experts whose opinions carry sway in the modeling industry, giving more legitimacy to the model.

The TMIP process does not provide a rubber stamp approval for a model. It involves objective and critical assessments of complicated technical details of the model. While the process is a tool to improve the model, participation does not imply Federal approval or endorsement of the model or its output.

Some interviewees suggested that it would be useful for TMIP to develop a standard PowerPoint presentation or memorandum that agency staff could customize and use as a tool to help communicate to senior managers and policy-makers the benefits of a peer review and to set realistic expectations on what it can and cannot do.

Provide more guidance and technical assistance to agencies for hosting a peer review.

Planning and hosting a peer review is a time-consuming, lengthy, and sometime complicated process that generates extremely useful results. By creating outreach material with tips on planning a peer review, TMIP could help smooth out the planning process for host agencies. Topics could include:

Choosing the stage in the modeling process: Description of factors to consider when choosing the point in the modeling process that the peer review would be most helpful.

Establishing the goals and purpose: Advice on how to determine the specific problems to address in the peer review and the specific charge to the panelists.

Recruiting panelists: Suggestions on how to select panelists along with developing a list of self-selected peer panelists and each person's expertise and experience with peer reviews.

Setting length and determining the schedule: Advice on how to decide the length and schedule of peer review meetings. This information could include comments on length and scheduling from agencies that have hosted peer reviews. It could also include reminders of practical matters such as how much time to request of the volunteer panelists.

This outreach material could be developed using ideas and advice from agencies that have conducted peer reviews. The material could include "lessons learned" and "best practices." The outreach material could include hard copies for distribution at conferences and elsewhere and be available on the TMIP website.

Conduct pre- and post-panel conference calls with panelists.

TMIP staff should consider scheduling a pre-panel conference call or webinar with the panelists and agency modeling staff. This call would allow the host agency to reinforce its modeling concerns and topics it wants the panel to address. It would also provide panelists the opportunity to ask host agency staff about technical modeling details. Logistical details such as invitational travel and the location of agency's office could also be addressed in the call.

A post-panel conference call or webinar could also be useful to conduct after participants have had some time to absorb the material presented at the meeting. Questions that arise after the meeting could be discussed. The note taker should participate in this call to provide details that the agency and the panelists might have missed.

Identify and inform panel of agency constraints prior to the meeting.

TMIP staff should work to ensure that the peer review does not produce a list of recommendations that the agency is unlikely to implement because of resource constraints, priorities, or policy issues. The agency should identify up front its limitations so that all peer review participants have a realistic notion of the agency's capabilities and so that recommendations are realistic and actionable under the agency's circumstances.

Provide more technical details in the post-panel report.

The post-panel report should have detailed technical information on the reasons for making specific recommendations. Agencies frequently refer back to the report when implementing changes. Additional technical details would be helpful to understand the reasons for specific recommendations. TMIP staff could assist this effort by providing clear guidelines on the expectations for the content in the report, particularly as it relates to the technical discussion. Technical discussion might be most appropriate in appendices or using hyperlinks so that details are available while maintaining the accessibility of the report for multiple audiences.

TMIP staff should consider providing technical assistance to agencies to help them prioritize peer review recommendations and develop an action plan for implementation.

Create a database that includes agencies and the issues discussed in their peer review for other agencies to use in planning peer review meetings.

Agency staff could use this database to identify organizations that have discussed similar topics at their peer reviews. Although the peer review summary reports are accessible on line, a database could be useful in narrowing the search to specific agencies for specific topics.

Provide agencies suggestions and advice for how to prepare a Request for Proposals (RFP) or interview consultants.

TMIP could provide advice to agencies that are issuing RFPs for travel demand models. This would help agencies develop RFPs that are technically accurate and have sufficient detail. Assistance could include:

Tips on how to develop a scope of work.

Recommendations for developing budgets.

Interview questions to use when hiring consultants.

Continue to conduct periodic evaluations to monitor the effectiveness of the peer review program.

Monitoring the program enables TMIP staff to identify elements that are working and where improvements are needed. It is also an effective way to see whether agencies are following through on implementing recommendations or not and the reasons behind their actions. In addition, TMIP should require host agencies to produce periodic status reports on their activities related to the peer review.

Continue to develop and disseminate materials on "state of the art" and "state of the practice" in travel demand modeling.

Because travel demand modeling is very technical, it can be difficult for agency staff to keep current with its latest developments. TMIP should continue to provide educational materials to help modelers understand the current trends in travel modeling and how agencies are implementing new tools.

Continue to work with modeling agencies to build peer networks.

As the peer review program has demonstrated, learning from peers has been valuable to agencies, modelers, and peer panelists. TMIP should continue to develop and promote networking opportunities to take advantage of formal and informal opportunities to learn from peers.

V. Conclusion

The results of this evaluation indicate that the TMIP Peer Review Program offers an effective process for agencies to improve their travel demand models. Agencies found the program to be very valuable for improving travel modeling in their regions. The process was instrumental in providing a valuable forum for them to identify and address important regional model components. When asked what type of agency would benefit from the TMIP Peer Review Process, interviewees indicated that it would be good for any agency involved in travel modeling, particularly MPOs, transit agencies, and state departments of transportation.

Appendix A: Interview Questions

Evaluation Questions for Evaluation of TMIP Peer Review Program

What were the key drivers that led you to request a peer review?

What were the major outcomes you were hoping for from the peer review?

How effective was the peer review in achieving these outcomes? (scale 1-5; please explain: What worked well? What did not work well?)

Did you make any changes to your model or modeling process based on the recommendations of the peer review panel? If so, what were the changes? Are you satisfied with the results?

Are there recommendations that you did not implement? If so, why not?

How well suited were the panelists in addressing your needs? (scale 1-5, please explain: What worked well? What did not work well?)

How effective was the meeting format in addressing your objectives? Would you recommend any changes to the format? (scale 1-5, please explain: What worked well? What did not work well?)

What suggestions do you have for improving the effectiveness of the various components of the peer review program? (e.g., the application process, choice of panelists, peer review session, peer review report, etc.)

What advice do you have for other agencies interested in participating in the peer review process? What agencies would profit most from participation? What can agencies do to ensure that they receive the most value from participation?

Appendix B: Description of Peer Review Meetings

The following is a summary of the peer reviews held by the agencies included in this evaluation.

The Baltimore Metropolitan Council (BMC) held two peer review meetings. The first focused on the status of the BMC travel model improvement process and provided guidance on near-term and long-term model development issues including demographic forecasting, incorporating new model functions such as managed lanes and truck traffic, and proposed new mode choice model. At the second peer review meeting, presentations and discussion focused on BMC's work plan to implement changes to its model based on the panel's recommendations from the first meeting. Both meetings were held at the office of the BMC in Baltimore, Maryland, with the same panel members.

The TMIP peer review was requested by the Baltimore Regional Transportation Board (BRTB), the MPO for the Baltimore metropolitan region. BMC staff provides support to the work activities of the BRTB. The Baltimore metropolitan region has a population of about 2.5 million people and covers an area of approximately 2,260 square miles.

The Baltimore model utilizes the traditional four-step process of travel demand forecasting. TP+ software (version 3.0) is used for running the model, except for the transit pathbuilding mode TRNBUILD. The purpose of the peer review was to provide the decision-makers in the region the opportunity to have the model reviewed by a panel of experts to determine how the model could be improved to meet future policy needs. Specifically, BMC asked the peer review panel to:

Evaluate the effectiveness of the current model.

Identify enhancements to the model that could reasonably be made in the short term.

Identify possible long-term enhancements to the model.

Provide feedback on the current BMC model.

Provide suggestions for future model enhancements.

Major recommendations from the second peer review meeting included:

Revise the traffic analysis zones early in the model improvement process.

Continue to coordinate closely with the Metropolitan Washington Council of Governments on employment and population forecasts.

Ensure that any changes are compliant with New Start guidelines produced by FTA.

Consider adding demographic factors to help validate model outcomes.

Ensure that modeled speeds are reasonable compared to actual speeds.

In an August 2005 memorandum, BMC addressed its planned next steps to incorporate these action items in a future work plan. Our interview with BMC focused on the extent to which it had made progress in moving these recommendations forward.

2. COMMUNITY PLANNING ASSOCIATION OF SOUTHWEST IDAHO (COMPASS)

The Community Planning Association of Southwest Idaho (COMPASS) is the MPO for northern Ada and Canyon counties of Idaho, including the city of Boise. The area has a population of approximately 481 thousand people. At the time of the meeting, COMPASS had recently transitioned from a three-step to a four-step model, including a mode choice component that was borrowed and adapted from Wasatch Front Regional Council (Salt Lake City MPO). The purpose of the peer review was to provide COMPASS with feedback on the technical aspects of their travel demand model, including recommendations for components that needed improvement.

Recommendations included:

Improve employment data.

Include greater market segmentation.

Review and address the imbalances between attractions and productions.

Apply composite impedances during trip distribution with all trips distributed and subsequently fed to the mode choice model.

Use network travel times for both model development and application purposes.

DRCOG is the MPO for the Denver area. The agency's planning area has a population of 2.5 million, with an additional growth of one million people expected over the next 20 years.

DRCOG held two peer review meetings in Denver, Colorado with the same panel members. The first meeting focused on identifying key planning issues and developing a vision for a new modeling system for the Denver region to include comprehensive redevelopment of all transportation and land use modeling elements. At the time of this meeting, DRCOG was in the early phases of replacing its existing land use and travel demand model with a fully integrated modeling system. The purpose of the second peer review was to discuss the elements to be implemented for DRCOG's model update.

The panel was charged with helping the DRCOG project team identify approaches to the development of an integrated model. The model improvement project, named the Integrated Regional Model Project, is essentially complete. As part of this project, DRCOG completed a Travel Behavior Inventory Project, a $1.5 million travel/activity/ demographic survey of the region.

Recommendations from the first meeting included:

Develop the capability to examine key policy choices using the model. The determination of what information policy-makers need to make informed decisions can help determine the level of detail or richness that the model must maintain.

Achieve integration for the various model elements with consistent time, geographic, demographic, and behavioral scales.

Provide an integrated information base; the Geographic Information System (GIS) can be crucial for this.

Improve the validity and reliability of results from the modeling process.

DRCOG reconvened the peer review panel in April 2004 to address general model design, potential staging issues, and other practical considerations for model development based on recommendations from the previous meeting. The peer review panel recommendations included:

The region should pursue an integrated data approach.

The model area boundary should be expanded to include the entire economic and air quality area.

Additional work is needed on the land use model approach, but this can be the focus of a longer-term effort.

The region should take a "three-pronged" approach on land use, household transportation, and freight modeling.

4. EAST-WEST GATEWAY COUNCIL OF GOVERNMENTS (EWGCG)

The East-West Gateway Council of Governments (EWGCG) is the MPO for the St. Louis, Missouri, metropolitan area. It has a population exceeding 2.4 million people and includes three counties in Illinois and four counties in Missouri.

The primary purpose of the peer review was to provide guidance to the EWGCG on model fixes and enhancements for its compliance review. Specifically, the panel was asked to:

Assess the sufficiency of the model, especially validation criteria and mode choice constants.

5. NORTH JERSEY TRANSPORTATION PLANNING AUTHORITY (NJTPA)

The North Jersey Transportation Planning Authority (NJTPA) region encompasses an area of 4,200 square miles with a population of 6.5 million people. NJTPA is the MPO for the 13-county Northern and Central New Jersey region, including the city of Newark, comprised of 384 municipalities. It is the Nation's fifth largest planning region.

NJTPA requested that the panelists assess the current travel demand model and recommend both near- and long-term model enhancements including:

At the time of the peer review, NJTPA had a standard four-step transportation model using TRANPLAN software. However, one of the main purposes of the peer review was to receive comments on implementing a four-step CUBE/Voyager TP+ model with a special FORTRAN routine for mode choice, a process the agency had already begun. The panel recommended that the NJTPA keep its near-term focus on using the validated Phase I CUBE model for its conformity analysis and concentrate on improving basic model elements such as networks and travel times. Other recommendations made by the panel included:

Use the household data more extensively.

Carefully examine highway output speeds and compare them to currently collected data.

Ensure that the network is commensurate with the traffic analysis zone detail.

Adopt tour based modeling for work trips rather than using "work-based other."

Incorporate subarea studies.

Separate university trips as a separate trip purpose.

Use the current model for airport trips.

Generate total person trips for non-motorized allocation to increase sensitivity.

Further examine the sensitivities of the nesting coefficients to the logsum for both the current and the proposed models.

Use composite impedance for trip distribution, reduce the number of K factors, and limit and justify the use of K factors work trips.

The panel felt that SANDAG's current model is consistent with the state of the practice, and the biggest question for SANDAG as it moves forward is how much to invest in revisions to the existing model versus the development of a new activity/tour-based model. The panel's primary recommendations included:

Expand data collection efforts to include a workplace survey, additional use of two-day travel diaries, and better transit and auto speed data.

Modify trip generation rates so that they are based on households and persons rather than structure type.

Run all trip purposes to convergence in trip distribution.

Explain and document the nesting coefficients used in the interim mode choice model. The basis of the coefficients may not be sufficiently scientific to meet FTA New Starts criteria.

Pay special attention to transit network coding.

Reduce post-processing volume adjustments by using observed speeds rather than level-of-service/capacity calculations in the volume-delay function.

Add root mean square error checks to validate model volumes.

Add a fourth time period for mid-day (important for air quality modeling since ozone levels are highest during the warmest hours of the day).

7. SOUTHEAST MICHIGAN COUNCIL OF GOVERNMENTS (SEMCOG)

The Southeast Michigan Council of Governments (SEMCOG) is the MPO for Detroit, Michigan. The region has a population of approximately 4.9 million people.

SEMCOG requested that the panelists assess the current travel demand model and recommend both near-term and long-term model enhancements. SEMCOG has a relatively new TransCAD-based four-step travel demand model. The model includes consideration of external, commercial vehicle, and transit trips. It is based on a variety of data sources, including census information, a SEMCOG household travel survey, traffic counts, and a transit-on-board survey.

The first day of the meeting focused on SEMCOG presentations on the current travel demand model and its plans for model improvements. Overall, the panel felt that the current model represents the state of the practice, addressing time-of-day, commercial vehicle, and external trips well. For future model enhancements, the panel's recommendations included:

The Southern California Association of Governments (SCAG) hosted a total of three peer review meetings. The first two were held in November 2003 and in April 2004. Both meetings took place in Los Angeles, California, with the same panel members. The evaluation focused on the third meeting, held in January 2006, which had a different set of panelists.

The purpose of the third peer review meeting was for the panelists to examine SCAG's current model and assess model updates, including changes to the trip distribution, mode choice, and trip assignment phases. SCAG also requested that the panelists comment on the agency's plans for model improvement and recommend near- and long-term model enhancements as well as to review validation targets for all model components.

SCAG is the MPO for six Southern California counties in the Los Angeles metropolitan area. The region encompasses an area over 38,000 square miles and includes a population exceeding 18 million people. The current SCAG model follows the traditional four-step modeling structure with three ancillary models feeding into network assignment, including an external trip model, a regional airport demand allocation model, and a heavy duty truck (HDT) model. The panel felt that SCAG 's current four-step model is consistent with the state of the practice, particularly the plan for the freight model, the use of the "strategic work trip" purpose in trip distribution, and the use of four time periods in traffic assignment. It also indicated that SCAG has done well in data collection efforts and recommended that SCAG make its impressive survey datasets available to other agencies.

The TMIP peer panel made a number of recommendations for SCAG to consider, including:

Focus on near-term validation of the model.

Assess the commuter rail nesting structure in the mode choice model as part of the mode choice validation process.

Test the vehicle availability model, which might be too sensitive to the density variable.

Investigate using a destination choice configuration for trip distribution and adding model components for high-occupancy toll lanes and peak spreading.