The following is the established format for referencing this article:
Marmorek, D. and C. Peters. 2001. Finding a PATH toward scientific collaboration: insights from the Columbia River Basin. Conservation Ecology 5(2): 8. [online] URL: http://www.consecol.org/vol5/iss2/art8/
A version of this article in which text, figures, tables, and appendices are separate files may be found by following this link.
Report

Finding a PATH toward Scientific Collaboration: Insights from the Columbia River Basin

Observed declines in the Snake River basin salmon stocks, listed under the U.S. Endangered Species Act (ESA), have been attributed to multiple causes: the hydrosystem, hatcheries, habitat, harvest, and ocean climate. Conflicting and competing analyses by different agencies led the National Marine Fisheries Service (NMFS) in 1995 to create the Plan for Analyzing and Testing Hypotheses (PATH), a collaborative interagency analytical process. PATH included about 30 fisheries
scientists from a dozen agencies, as well as independent participating scientists and a technical facilitation team. PATH had some successes and some failures in meeting its objectives. Some key lessons learned from these successes and failures were to: (1) build trust through independent technical facilitation and multiple levels of peer review (agency scientists, independent participating scientists and an external Scientific Review Panel); (2) clarify critical uncertainties by developing common data sets, detailed sensitivity analyses, and thorough
retrospective analyses of the weight of evidence for key alternative hypotheses; (3) clarify advice to decision makers by using an integrated life cycle model and decision analysis framework to evaluate the robustness of potential recovery actions under alternative states of nature; (4) involve key senior scientists with access to decision makers; (5) work closely with policy makers to clearly communicate analyses in nontechnical terms and provide input into the creation
of management alternatives; and (6) recognize the trade-off between collaboration and timely completion of assignments.

Salmon populations in the Snake River sub-basin of the Columbia River in Oregon and Idaho, USA (Fig. 1) have been in decline since the late 1800s (National Marine Fisheries Service 1995), with particularly dramatic declines occurring since the mid-1970s (Schaller et al. 1999). Snake River spring, summer, and fall chinook (Oncorhynchus tshawytscha), and steelhead (Oncorhynchus mykiss) are all listed as threatened under the Endangered Species Act (ESA; see Table 1 for definitions of acronyms), and Snake River sockeye (Oncorhynchus nerka) are listed as endangered. Accelerating declines have been attributed to various factors: habitat degradation in freshwater and estuary rearing areas; tributary and ocean harvest (particularly for fall chinook and steelhead); interaction with hatchery steelhead and chinook smolts; changes in ocean conditions; completion in 1976 of the eight dams and reservoirs that comprise the Federal Columbia River Power System (FCRPS); and initiation of large-scale transportation of smolts in trucks and barges from the upper dams past the FCRPS to below Bonneville Dam (the last dam encountered by smolts as they migrate out of the Columbia River).

Fish Leaving Under Several Hypotheses (passage model developed by States and Tribes)

FCRPS

Federal Columbia River Power System

IDFG

Idaho Department of Fish and Game

ISAB

Independent Scientific Advisory Board

ISRP

Independent Scientific Review Panel

I. T.

“Implementation
Team” (Regional Forum for the Implementation of the 1995 Biological
Opinion)

NEPA

National Environmental Policy Act

NMFS

National Marine Fisheries Service

NPPC

Northwest Power Planning Council

ODFW

Oregon Department of Fish and Wildlife

PATH

Plan for Analyzing and Testing Hypotheses

SRP

Scientific Review Panel for PATH

USFS

U.S. Forest Service

USFWS

U.S. Fish and Wildlife Service

USGS-BRD

U.S. Geological Survey-Biological Resources Division

WDFW

Washington Department of Fish and Wildlife

Simultaneous changes in the system confound attempts to determine the importance of the four H’s (habitat, harvest, hatchery, and hydro) and ocean conditions in causing historic stock declines (Fig. 2), and, consequently, the best course of action for recovering stocks. Different beliefs about which factors contributed most to observed declines resulted in the development of three different modeling systems for evaluating recovery strategies for Columbia River salmon stocks. These systems were developed by State and Tribal fishery
managers, federal hydropower operating agencies (Bonneville Power Administration (BPA), the U.S. Army Corps of Engineers, and scientists at the Northwest Power Planning Council (NPPC), a political body that oversees research programs in the Columbia River. The models used the same basic kinds of information, but had different underlying assumptions that generally reflected the policy favored by the sponsoring agency. As a result, they provided different, often-conflicting management advice about the relative performance of alternative recovery strategies for Snake River populations.

Fig. 2. Confounding changes in factors affecting Snake River salmon over the last four decades: number of dams passed by Snake River salmon (Schaller et al. 1999); total hatchery releases in Snake River Basin (Williams et al. 1998); historical fraction of spring/summer chinook transported in barges, as estimated by two different passage models, CRiSP and FLUSH (Marmorek et al. 1998b); and Pacific Decadal Oscillation (higher values associated with poorer ocean conditions for Snake River salmon; Mantua et al. 1997).

In 1995, the National Marine Fisheries Service (NMFS), after three years of comparing these models and subjecting them to external peer review, issued a Biological Opinion on the FCRPS (a document that summarizes the status of listed stocks and prescribes hydropower system operations to avoid extinction), in which they concluded that the emphasis should shift to identifying and evaluating the models’ assumptions (NMFS 1995:124, Rec. 17). This recommendation was augmented by a 1994 court ruling (IDFG vs. NMFS, D.Or 1994) that determined that NMFS must consult with State and Tribal biologists. The new, collaborative process that was formed in response to the NMFS and court recommendations became known as the Plan for Analyzing and Testing Hypotheses (PATH). At its formation, PATH was intended to help reduce uncertainties in NMFS's future hydrosystem decisions.

PATH operated from September 1995 to May 2000 at a total cost of U.S.$7 million, funded from BPA power revenues through the NPPC’s Fish and Wildlife Program. We had the challenging tasks of facilitating and coordinating the PATH process. Our purpose in this paper is to summarize the overall PATH process, the complex institutional context in which it operated, and lessons learned that might apply to other collaborative research processes. Although we have considered the perceptions of others working within PATH or using its results, the conclusions
presented here are our own.

WHAT SCIENTIFIC AND POLICY CHALLENGES EXISTED BEFORE PATH? WHAT ANALYTICAL PROCESS WAS REQUIRED TO OVERCOME THEM?

Columbia River agencies faced many institutional and technical challenges to selecting the best long-term recovery strategy (see column 1 of Table 2). Many of these challenges are common in resource management problems. Less common are analytical processes with the necessary attributes to meet these challenges and help managers make difficult decisions (see column 2 of Table 2). Although no process is likely to meet all of these requirements, the degree to which these requirements were met in PATH forms a useful template for assessing its strengths and weaknesses.

Table 2. Pre-PATH challenges, associated requirements of an
analytical process, and the PATH approach used to meet those requirements.
Note that some of these requirements conflict (e.g., building trust
vs. enforcing deadlines), whereas others are complementary.

Decision-making challenges

Requirements of analytical process

PATH approach

1. Differing objectives among entities

receive clearly defined management
objectives from policy groups

alternative actions to be evaluated
should capture range of objectives of participating agencies

external
oversight

prospective analyses

2. Low level of trust among scientists and agencies;
many court cases

build trust and cooperation through
meaningful participation of representative range of agency and
other scientists

process and its scientists should
be isolated from political influences, yet focused on pressing
decisions

internal
structure

internal and external review

3. Lack of understanding of differences in model’s
underlying assumptions

use a single integrative data
and modeling framework to assess alternatives and provide scientifically-defensible
advice

clearly communicate technical
analyses to nontechnical audiences

internal
and external review

prospective analysis

5. Urgency of decision

enforce deadlines and ensure timely
delivery of products

moral
suasion

WHAT TOOLS AND APPROACHES DID PATH USE TO MEET THESE CHALLENGES?

PATH adopted six tools and approaches to meet these scientific and policy challenges (see column 3 of Table 2).

External oversight

The structure of PATH resulted from discussions with senior policy makers and scientists in 1995. PATH’s immediate “client” was the Regional Forum for the Implementation of NMFS’s 1995 Biological Opinion, or “Implementation Team” (I. T.) (Fig. 3). The Implementation Team included Federal and State fish and wildlife agencies and FCRPS operating agencies. Although the I. T. represented a broad range of interests and objectives, its mandate was to focus entirely on hydrosystem operations and associated long-term decisions. About four times per year, the I. T. received detailed presentations on PATH’s findings and helped to prioritize its activities.

Fig. 3. Overall structure of PATH, and agencies contributing scientists to the process. See Table 1 for definitions of agency acronyms.

Internal structure

PATH consisted of about 30 scientists who contributed directly to workshops, analyses, and products (Appendix 1). Specialties included fisheries biology and management, analytical and modeling skills, field studies, experimental design, and dam operations. Although most scientists were employees or consultants from 12 regional institutions (Fig. 3), PATH also included an independent facilitation team and three independent scientists. The three independent scientists, who had expertise in the areas of conservation biology, quantitative methods, fisheries management, and decision analysis, assisted participants in developing and critiquing analytical methods and results. PATH activities were coordinated by a six-member Planning Group representing the facilitation team, State, Tribal, and Federal fishery agencies, the power system operating agencies, and the NPPC (Northwest Power Planning Council).

The role of the facilitation team was to stimulate and organize PATH analyses, provide clarity to decision makers, maintain the creativity and commitment of PATH scientists, and ensure that analyses met the highest scientific standards. Given the turbulent history, a major challenge was to build trust (or at least mutual respect) among scientists from competing agencies. We used five methods to do this: (1) establish ground rules for interactions (e.g., “be hard
on the problem, easy on the people” Fisher et al. 1991); (2) redirect personal attacks into examinations of evidence for alternative hypotheses; (3) develop structured approaches that permit meaningful participation by all PATH scientists; (4) formally document evidence for and against alternative hypotheses; and (5) demonstrate qualities we wished others to display (i.e.,
respect, objectivity, integrity, creativity, and humor).

Internal and external review

The PATH process was deliberately iterative, with four levels of peer review (Fig. 4). Small analytical work groups generally completed initial drafts of analyses, which were then reviewed and refined by larger work groups. Ultimately, all PATH participants reviewed draft reports, and the PATH Scientific Review Panel (SRP) and other regional scientists provided external review of final products. The SRP consisted of four independent scientists providing arm’s length peer review. They spent 150 days over four years reviewing about two thousand pages of PATH reports, and provided valuable direction on methods and priorities for future work.

Fig. 4. Structure of work flow and peer review within the PATH process.

PATH products were available to the public through open presentations to the I. T. (Implementation Team) and NPPC, and a dozen PATH scientists gave a special joint presentation to the public in February 1999. PATH's presentations were frequently reported by regional print and electronic media, and all reports were made available on the BPA web site http://www.bpa.gov/Environment/PATH. This provided an opportunity for review by other regional scientists.

Retrospective analyses

Retrospective analyses followed from the requirement to understand the fundamental differences between models, and set the foundation for prospective analyses (Fig. 5).

Fig. 5. Flow of activities under PATH’s three objectives of retrospective analyses, prospective analyses, and experimental management. PATH did not have the opportunity to complete all of the activities in the right-most box.

The retrospective analyses:
1) identified the assumptions underlying different modeling systems, and their management implications;
2) expressed these assumptions in terms of unambiguous hypotheses about historical stock trends and causal factors;
3) assessed the level of support for alternative hypotheses using existing data; and
4) identified information gaps that constrained the ability to distinguish among alternative hypotheses.

Hypotheses for PATH retrospective analyses of Columbia River spring/summer chinook stocks were developed using a three-level framework to address observed patterns in specific life history stages and stressors that act on those stages (Fig. 6; Appendix 2). Assessments for Snake River steelhead and fall chinook were less comprehensive than for spring/summer chinook because of data limitations.

Fig. 6. Three-level hypothesis framework used to guide retrospective analyses (Marmorek et al. 1996a). Level-3 analyses need to be consistent with Level-1 and Level-2 analyses. The figure shows only some of the linkages between management actions and life history stages.

Prospective analyses

PATH prospective analyses evaluated the biological consequences of three primary hydrosystem recovery actions for ESA (U.S. Endangered Species Act)-listed Snake River salmonids: continue current operations (option A1); maximize transportation of salmon smolts down the migration corridor (A2); or natural river drawdown of four Snake River dams (A3). Information gained from the retrospective analyses was incorporated into the prospective analyses using a decision analysis approach (Fig. 5), which had been recommended by both the SRP and PATH independent scientists. The prospective/decision analyses of spring/summer and fall chinook employed
models to simulate outcomes for each hydrosystem action under a range of hypotheses about various uncertainties, expressed as a decision tree (Fig. 7). This
allowed us to: (1) look systematically at the outcomes of management actions across a range of hypotheses; (2) conduct a detailed sensitivity analysis of results to determine “key uncertainties” (Appendix 3); and (3) determine
which actions performed well over a broad range of uncertainties (i.e., were most robust). The simulations also considered the range of uncertainty in stock productivity parameters and climate conditions, as estimated from retrospective analyses (Deriso, in press). We calculated weighted average outcomes for each action and also examined the distribution of outcomes over all combinations of hypotheses. Initially, we weighted all hypotheses equally, but later developed a “Weight of Evidence process” for spring/summer chinook to elicit probabilities on hypotheses from the PATH SRP (Fig. 8, Appendix 4).

Fig. 7. Summary of general decision tree used in PATH. See Peterman and Anderson (1999) for a summary of decision analysis. Two or more alternative hypotheses were considered for each of the uncertainties listed at the bottom of the figure. Extra mortality and transportation hypotheses had the strongest effects on the ability of management actions to meet survival and recovery standards (Appendix 3). See Marmorek et al. (1998b, d), Peters et al. (1999, in press), and Peters and Marmorek (in press) for detailed results.

Fig. 8. Weight of Evidence approach used in PATH (Marmorek and Peters 1998c). This process used sensitivity analysis to identify key uncertainties, systematically organized the evidence for and against alternative hypotheses for these uncertainties, elicited independent judgments from the SRP (Scientific Review Panel) on the relative probabilities of alternative hypotheses, and examined the consequences of these probability judgments in the decision analysis. Although management experiments were recognized as a much more rigorous approach to assigning probabilities to alternative hypotheses, such experiments may take decades. The Weight of Evidence process provides decision makers with a method of making a decision now, given current uncertainties, and clearly documents the rationale for this decision. See Appendix 4 for more details.

Experimental management

Experimental management reduces key uncertainties by deliberately implementing spatial and temporal contrasts in treatments and associated monitoring. Although the SRP had recommended in 1996 that PATH design and evaluate experimental management approaches, this exploration only began in December 1998. The goal was to find management approaches that met conservation and recovery objectives, while also generating information to help select long-term management actions. PATH generated a set of seven experimental management and three research activities to distinguish among critical alternative hypotheses that implied different long-term decisions (Table 3). For each candidate activity, we described their spatial and temporal components, associated monitoring, potential learning benefits, risks to stocks, and practical constraints. In October 1999, this list was pared down by the I. T., who selected five experimental management actions plus a base case of current operations for further study of the quantitative trade-off between learning and conservation objectives (Appendix 5). This evaluation was completed in April 2000.

In this section, we assess the extent to which PATH was able to meet each of the challenges facing decision makers on the Columbia River, and some of the lessons learned from PATH’s successes and failures. We focus on some of the key lessons here and provide a longer list of lessons learned in Table 4.

Table 4. Summary of lessons learned from the
PATH experience.

Challenges

Lessons learned from PATH

1. Differing objectives among entities

Analysts should provide decision
makers with clear guidelines to ensure that alternatives can
be implemented in the modeling framework.

Process should allow scientists
to participate in developing alternatives.

Enlist expert reviewers for the
duration of the process to ensure that they have detailed knowledge
of the analysis.

Involve external scientists (different
from reviewers) in the analyses.

Independent facilitation.

3. Lack of understanding
of differences in model’s underlying assumptions

Thorough sensitivity analyses help
to build common understanding among scientists about key uncertainties.
Do these early in process before spending time and money on
resolving inconsequential uncertainties.

Common data sets made it easier
to identify where different analyses diverged.

Some uncertainties are unresolvable
short of deliberate management experiments; policy makersmust
be educated about the benefits of experimental management using
success stories.

Experimental management is easier
to sell/implement before species are ESA-listed.

4. Lack of clear advice
to decision makers

A common modeling framework elucidated
key uncertainties and assessed robustness of actions across
a range of assumptions (better than reconciling separate analyses
with different assumptions).

The complexity of the common framework
made it more difficult to understand internal workings of models
and communicate findings to nontechnical audiences.

Recognize trade-off between scientific
relevance and ease of explanation of performance measures.

Having a separate research institution
with scientists seconded from agencies allows more efficient
use of scientists time.

Recognize trade-offs between timely/relevant
reporting of results to decision makers, degree of collaboration,
and publishing in peer-reviewed journals.

1. Differing management objectives among the participating entities

PATH generally received clear direction from the Implementation Team on management objectives (largely dictated by the Endangered Species Act) and what hydrosystem actions to analyze. The I. T. focused exclusively on hydrosystem actions, consistent with its mandate from NMFS’s 1995 Biological Opinion on the FCRPS, and because other agencies (e.g., U.S. Forest Service, U.S. Fish and Wildlife Service) were reluctant to allow the I. T. to venture into their jurisdictional territories by specifying non-hydrosystem (habitat, harvest, and hatchery) actions. PATH considered only hydrosystem decisions as actions, but we did include habitat, harvest, and hatchery effects as uncertain states of nature in the decision analysis or as sensitivity analyses. Ultimately, though, PATH was criticized because the actions that we evaluated did not adequately capture the complete range of management alternatives and objectives, and thus did not fully address the challenge of differing management objectives among agencies.

PATH scientists recognized the potential pitfalls of focusing only on hydrosystem actions, but were not involved in developing the alternatives and thus had no opportunity to affect the choice of actions. This suggests that analytical processes should allow scientists more input to the development of alternative actions and should broaden the mandate of policy groups like the I. T.

2. Lack of trust among scientists and agencies over past disputes and court cases with dueling models

Over PATH’s lifetime, interactions among PATH scientists became much more respectful and constructive. Communication of disagreements progressed from personal attacks to identification of alternative hypotheses and examination of evidence. This was due to several reasons. First, the use of decision analysis did not require consensus, but instead considered all tenable hypotheses and assessed their implications for management decisions. Sensitivity analyses
showed that decisions were insensitive to several alternative hypotheses that had been proposed, saving many hours of purposeless debate. Second, it was critical to have independent facilitators, who emerged as the only ones trusted to prepare, revise, and present final PATH products. In an analytical process such as PATH, the facilitation team must be technically proficient so that they can understand the analyses and its assumptions, mediate technical arguments between parties, and present findings to nontechnical audiences. Third, the rigorous internal review process within PATH gave everyone an opportunity to see each other’s analyses before they became public. This eliminated the possibility of surprise attacks characteristic of previous court cases.
Fourth, the three independent scientists who actively participated in PATH analyses played a critical role in keeping debates within PATH scientifically grounded and ensuring that arguments over analyses and results were based on facts, not agency positions. Finally, the PATH SRP played a key role as an external arbiter of alternative hypotheses put forward by different scientists. Their involvement throughout PATH’s entire duration gave them an intimate understanding of the issues, models, and analyses. The long-term engagement of the PATH SRP differs from most Blue-Ribbon Panels of Experts, which generally do not have the time or resources to gain as in-depth an understanding of the models and data that they review.

PATH was partially successful in involving a representative range of scientists. Oversight by the multi-agency Implementation Team ensured involvement of a wide range of agencies and interests, and prevented any single agency from overly influencing PATH. Independent funding for PATH participants through the NPPC Fish and Wildlife Program also helped to ensure a level playing field. However, PATH was ultimately unsuccessful in actively involving key senior-level scientists from the NMFS Northwest Fisheries Science Center in Seattle, Washington, USA. In the Pacific Northwest, the Science Center has a very strong influence on NMFS decisions on endangered species. Although three scientists from the Science Center occasionally participated in PATH, they each had numerous other responsibilities, and as a result had less comprehension and ownership of PATH’s methods and conclusions. NMFS was represented in PATH by three very competent scientists from their Portland, Oregon office, but ultimately these scientists had less influence within their agency than did Science Center staff. Had NMFS required senior scientists from their Science Center to participate more directly in PATH starting in 1996, the process and methods probably could have evolved to incorporate their concerns, while still retaining an integrative framework. Instead, the NMFS Science Center eventually developed their own tools and analyses through their Cumulative Risk Initiative (CRI), which ultimately had more influence than PATH on NMFS’s decisions on the hydrosystem. This experience emphasizes the need to involve influential key scientists with strong links to ultimate decision makers.

3. Lack of understanding of differences and similarities in the models’ underlying assumptions

PATH retrospective analyses successfully elucidated differences between models, brought substantial empirical information to bear on alternative hypotheses to explain recent declines, produced considerable convergence on historical data sets (some of which were previously thought to be unusable, such as the spawner-recruit data), and made a significant contribution to the regional data inventory. The retrospective analyses also identified major uncertainties in past and current conditions that were unresolvable because of incomplete data and historical confounding (Fig. 2); these uncertainties were carried forward into the prospective decision analysis.

The decision analysis framework permitted rigorous sensitivity analyses of how each uncertainty affected the ranking and performance of actions. Importantly, we found that the different models’ estimate of the survival rate of in-river migrants through the hydropower system, a hotly debated value, was NOT an important determinant of overall life cycle survival. Rather, the key uncertainties that emerged from these sensitivity analyses were related to the cause of mortality in the estuary and ocean (Appendix 3). Key uncertainties were assessed through the Weight of Evidence process (Appendix 4), in which the PATH SRP judged arguments for and against alternative hypotheses and assigned probabilistic weights.

PATH had only limited success in identifying research, monitoring, and adaptive management actions to resolve uncertainties. Although the SRP highlighted the importance of experimental management work in 1996, the I. T. did not authorize PATH to focus seriously on this issue until 1999. We believe that the I. T. placed low priority on experimental management for a number of reasons. First, 1999 deadlines for important regulatory decisions pressured agencies to make firm long-term decisions (with supposed omniscience), rather than acknowledging uncertainty and encouraging deliberate management experiments. Second, discussions of experimental management actions made policy makers uneasy because of their inherent uncertainty. They correctly perceived the risk that management experiments may not deliver desired survival improvements, but were not swayed by arguments that implementing long-term decisions now would also have uncertain results and generate less information than a well-designed experiment.

In the end, PATH’s experience with experimental management was similar to that of others who have run into barriers to the implementation of adaptive management (Gunderson et al. 1995, Walters 1997, and MacDonald et al. 1999), and continues a tradition of ultra-passive approaches to adaptive management within the Columbia Basin (McConnaha and Paquet 1996). Perhaps testimonials from other jurisdictions where experimental management approaches were successfully used would have helped. However, the main lesson here is that experimental management is much easier to sell to policy makers, and to implement, before populations are placed on the endangered species list. Afterward, statutory restrictions on management actions, political pressure for immediate action, public attention on policy decisions, and the precarious status of the stocks make it almost impossible to implement experimental management.

4. Lack of clear advice from scientists to decision makers

PATH had mixed success in providing clear management advice to decision makers. We were successful in developing a single integrative data and modeling framework by melding three different life cycle models into a single Bayesian Simulation Model (Deriso in press;Appendix 6). By developing common data sets, an integrated modeling framework, and detailed sensitivity analyses, PATH greatly clarified the overall effects of different assumptions and provided a common understanding of the factors that determined the performance of recovery actions. This was a major advance over pre-PATH approaches, which had to reconcile results of different modeling systems using different data and assumptions.

PATH was less successful in clearly communicating technical analyses to nontechnical audiences. Part of this was because of the relatively complex structure of the integrative modeling framework, which had to be flexible enough to accommodate multiple hypotheses. The integrative life cycle model and the decision analysis, which incorporated multiple hypotheses, were critical approaches for building trust among PATH participants and understanding the factors affecting model results and performance of actions. However, these methods were hard to explain to nonscientists having little experience with these types of risk assessment methods. PATH’s performance measures (the 1995 Biological Opinion Jeopardy Standards) were also complex. It often proved to be difficult to communicate to nontechnical audiences “the probability of the geometric mean number of spawners exceeding a recovery threshold over 4000 simulations.” PATH reports were carefully worded to ensure that the views of all participants were fairly represented. As a result, PATH reports became comprehensive and
technically demanding documents that were ill suited for nonspecialists.

A lesson here is that scientists need to think creatively about how to communicate summaries of risk assessments to nontechnical audiences, and to allocate sufficient time and personnel (e.g., technical editors, graphic designers) for producing such summaries. Thoughtfully designed interactive models can also provide nontechnical audiences with insights into the nature of variability and uncertainty (Walters 1994). Another lesson relates to the trade-off between the scientific relevance of model performance measures and their complexity. The Jeopardy Standards
were meaningful to PATH scientists, but were difficult to explain to the public. It may have been wise to develop a simpler set of performance measures that, although perhaps less comprehensive, were more readily understood by nonscientists.

5. NMFS was under pressure to decide quickly

PATH was usually several months late in delivering results. This was largely because of the time required for participating scientists and agencies to agree on basic data sets, integrate models into a decision analysis framework, complete four levels of peer review, and agree on precise wording of Executive Summaries and presentations. As technical facilitators and synthesizers, we had no formal authority to pressure participants to deliver data and analyses on time, but instead relied on moral suasion to enforce deadlines. A more efficient approach might be to have scientists seconded from their agencies to serve on a task force under some independent institution, and/or give the leadership of such processes strong authority to enforce deadlines (scientific committees of the Intergovernmental Panel on Climate Change do the latter).

CONCLUSION

In December 2000, NMFS released their final Biological Opinion on the FCRPS (NMFS 2000). This plan largely maintains status quo operation of the hydropower and transportation systems, and relies on mitigation through habitat, harvest, and hatchery management to generate necessary survival improvements. In their decision, NMFS relied almost exclusively on analyses conducted by its Northwest Fisheries Science Center. Although some elements of the PATH analyses contributed to the structure of NMFS’s internal analyses (e.g., spawner-recruit data set, emphasis on uncertainties in the estuary and ocean life stage), NMFS did not accept PATH’s primary conclusions.

Why did PATH ultimately have less influence on NMFS’s decision than was originally envisioned? Given the issues at stake, NMFS was probably reluctant to commit to a collaborative scientific process whose methods, outcome, and schedule they could not control. Both NMFS and the NPPC faced political pressure from various groups to avoid or defer a drawdown decision, which PATH’s results generally supported. Other factors include PATH’s failure to
actively involve senior, influential scientists within the NMFS Northwest Science Center, and an insufficient effort to communicate its results to senior policy makers within NMFS and the NPPC. Without a high level of involvement in, or ownership of, PATH analyses, these influential agencies perceived that they could have more control over the analyses and the decision by conducting their own internal analyses. These weaknesses ultimately undermined PATH’s funding support.

The PATH experience thus demonstrates some of the difficulties in implementing
collaborative analytical approaches, but it also demonstrates the value of such
an approach. Prior to PATH in 1993-1994, NMFS was faced with half a dozen lawsuits.
While PATH was operating from 1995 to 2000, there were no lawsuits initiated
against NMFS by the participating agencies. Since the termination of PATH in
May 2000, many state, tribal, and environmental groups have expressed their
dissatisfaction with the lack of collaboration by NMFS in the development of
their December 2000 Biological Opinion. It therefore appears unlikely that the
parties will remain outside of the courtroom for long in the absence of an established
collaborative process. We believe that the courts are not the best place for
scientists to do their work of testing alternative hypotheses and assessing
alternative actions in the context of multiple uncertainties.

RESPONSES TO THIS ARTICLE

Responses to this article are invited. If accepted for publication, your response
will be hyperlinked to the article. To submit a comment, follow
this link. To read
comments already accepted, follow this link.

Acknowledgments:

The authors thank PATH participants, the SRP (Scientific Review Panel), and members of the I.T. for their commitment, creativity, and hard work over five years. Randall Peterman and two anonymous reviewers provided helpful comments on an earlier draft. This research was funded by the Bonneville Power Administration, as part of the NPPC (Northwest Power Planning Council) Fish and Wildlife Program.

Larry Barnthouse, LWB Environmental Co. (Coordinator)
Steve Carpenter, University of Wisconsin
Jeremy Collie, University of Rhode Island
Brian Dennis, University of Idaho
Saul Saila, University of Rhode Island
Carl Walters, University of British Columbia

PATH developed a three-level hypothesis framework for structuring retrospective analyses of spring/summer chinook stocks (Fig. 6). The analyses were published in Marmorek et al. 1996a,1998a, c, and were summarized in Marmorek et al. 1996b. By explicitly embracing the notion of multiple working hypotheses (Chamberlain 1890), PATH went beyond most fisheries stock assessments conducted by single agencies.

Level-1 hypotheses sought to identify differences in trends among species/stocks, but did not propose mechanisms to explain those differences. Level-1 analyses completed by PATH included analyses of broad geographical and temporal trends in stock indicators such as recruits, spawners, and recruits/spawner (Deriso et al. 1996, in press, Schaller et al. 1999, Botsford and Paulsen 2000). An additional contribution of these analyses was the development of a comprehensive set of spawner-recruit data for 13 spring/summer stocks from the Snake River and other areas of the Columbia River basin (Beamesderfer et al. 1997). These analyses concluded that, while productivity and survival rate of all Columbia River spring/summer chinook stocks declined between the pre-1970 and post-1975 periods, Snake River and Upper Columbia stocks showed steeper patterns of decline over this time period than did lower Columbia stocks.

Level-2 hypotheses sought to explain trends in stock indicators during particular life history stages in terms of spatial/temporal contrasts in survival and candidate stressors that might be historically correlated with these survival patterns (i.e., hydropower, habitat, hatchery, harvest, and climate). Here the intent was to provide inferences from historical data on where to focus future management actions, in terms of both life history stages and stressors, and to elucidate gaps in the information required to distinguish among competing hypotheses. These analyses focused primarily on spring/summer chinook, including: regression analyses of recruits per spawner against various stressors (Paulsen 1996); assessments of the influence of spawning and rearing habitat (Petrosky and Schaller 1996, Paulsen et al. 1997, Petrosky et al. 2001); various analyses of the correlation between hatchery releases and survival/mortality rates (Wilson 1996, Budy et al. 1998, Paulsen and Hinrichsen 1998, Petrosky 1998, Williams et al. 1998); analyses of the role of climatic changes on salmon production (Anderson 1996, Hinrichsen et al. 1997, Paulsen and Fisher 1997); analyses of harvest impacts (Langness et al. 1998); and assessments of the effects of the hydrosystem (Deriso et al. 1996, in press, Schaller et al. 1999). Data sets developed for these analyses include survival indices for two phases of the spring/summer chinook life cycle: parr-smolt (Paulsen et al. 1997) and spawner-smolt (Petrosky and Schaller 1996). Major conclusions from Level-2 analyses were that hydrosystem, habitat, and climate conditions have all contributed to observed patterns of decline in Snake River stocks (although statistical power analyses showed that habitat changes alone were not sufficient to explain these patterns). PATH concluded that harvest effects did not contribute significantly to post-1974 declines, and that hatchery programs were probably also not a major factor.

Level-3 hypotheses sought to explain the specific mechanisms associated with observed trends in each life history stage identified at Level 2. These hypotheses link directly to key management decisions, which are affected by the quantitative magnitude of various effects (e.g., changes in survival with increased flow) rather than whether or not an effect merely exists. PATH completed a wide range of Level-3 analyses for spring/summer chinook and fall chinook, and a limited set of analyses for steelhead and sockeye. These analyses focused on the effects of specific hydrosystem actions on survival through the juvenile migratory corridor. For example, a PATH subgroup developed a detailed flowchart of the expected response of juvenile spring/summer chinook salmon to various
operations and configurations of dams in the Snake and Columbia River (Toole et al. 1996). An important data set for Level-3 analyses was the mark–recapture experiments with PIT (Passive Induced Transponder) —tagged juvenile salmon conducted by NMFS (e.g., Muir et al. 1996). These data were useful for estimating reach- and project-specific survival rates of migrating salmon smolts, and for estimating smolt-adult survival rates of
transported and nontransported fish. PATH conclusions from Level-3 hypotheses were that modifications to the existing hydropower system were not likely to improve juvenile survival rates. Transportation of smolts improves the direct survival of smolts, but there was insufficient information about delayed effects of transportation to say whether transporting smolts improves overall spawner-to-recruit survival rates. In addition, drawdown of Snake River dams can compensate for effects of the hydrosystem and improve juvenile survival rates.

The results of these retrospective analyses were ultimately condensed into a clearly written, 30-page Conclusions Document (Marmorek et al. 1996b). The tough internal and external review process led to 10 drafts of this document, but ensured that the strength of the conclusions was consistent with the available
evidence.

FALL CHINOOK

Retrospective analyses for fall chinook were less comprehensive than for spring/summer chinook because of time constraints and limitations in available data (Peters et al. 1999, in press). For example, for fall chinook there is a shorter time series of spawner-recruit and juvenile passage survival data. Retrospective analyses for fall chinook generally focused on the consistency of different structures of stock-recruit models with spawner-recruit data developed for four fall chinook stocks from the Snake, Lewis, and Deschutes Rivers, and from the Hanford reach of the Columbia River. These models embodied different hypotheses about the importance of transportation, hatchery supplementation, and climate effects on historical trends in survival rates (these analyses were roughly analogous to Level-2 analyses in the spring/summer hypothesis framework). We also explored overall trends in spawner-recruit survival (analogous to Level-1 analyses), and reviewed evidence on the survival of smolts through various components of the hydrosystem such as survival through turbines and in spillways (analogous to Level-3 analyses).

APPENDIX 3Key Uncertainties in the PATH Decision Analyses

KEY UNCERTAINTIES

PATH conducted extensive sensitivity analyses to determine which uncertainties were most influential in determining model outcomes. This involved comparing outcomes for all of the runs (i.e., all combinations of hypotheses) with the outcomes under a subset of runs associated with a particular hypothesis, or a particular combination of hypotheses. These comparisons looked at both the ability to meet survival and recovery standards and the ranking of decisions (i.e., which action had the higher probability of meeting a given standard). We considered a range of differences in these probabilities (i.e., 0.02, 0.06, and 0.1) to assess how robust our conclusions were. To independently check the inferences we drew from these methods, we applied Categorical Regression Tree analysis to the complete data set of decision analysis outcomes. The ‘CART’ trees clearly showed the relative importance of each action and hypothesis to the computed probabilities of survival and recovery. Sensitivity analyses for spring/summer chinook are described in Marmorek et al. (1998b, c, d) and Peters and Marmorek (in press). Sensitivity analyses for fall chinook are described in Peters et al. (1999, in press).

Using this approach, PATH scientists identified two key uncertainties that have the
strongest effects on survival and recovery of Snake River spring/summer and
fall chinook: extra mortality of nontransported fish and the relative
post-Bonneville survival of transported fish compared to post-Bonneville
survival of nontransported fish. Bonneville Dam is the last of eight dams that
smolts pass on their way to the ocean (“BON” in Fig. 1).

1. Extra mortality of nontransported fish

Extra mortality is defined as any mortality occurring outside the juvenile migration corridor that is not accounted for by the other terms in the life cycle model used for retrospective and prospective modeling (i.e., terms for stock productivity and carrying capacity, mortality in dams and reservoirs, and estuarine/ocean mortality affecting all salmonid populations). Because many of the changes that may account for historical patterns in extra mortality all happened around the same time (e.g., Fig. 2) there is uncertainty about which of these factors (or mix of factors) influences extra mortality. Therefore, PATH formulated three alternative hypotheses about the source of this extra mortality:

a. Hydro – extra mortality is related to the experience of smolts that pass through the hydropower system (e.g., delayed effects of stress).

b. Regime shift – extra mortality follows a 60-yr cycle that is related to long-term cycles in ocean conditions. There are no actions that can be taken
to reduce extra mortality, but extra mortality will eventually go down when
ocean conditions improve.

c. Stock viability (here to stay) – extra mortality is due to some phenomenon that will not be affected by any hydrosystem action or regime shift (i.e.,
interaction with hatchery fish, presence of diseases such as Bacterial Kidney
Disease, or reduction in nutrients associated with historical declines in
spawning stock).

Extra mortality can only be inferred from other measured quantities; it cannot be
directly measured. This makes it difficult to monitor changes in extra
mortality resulting from an experimental action, and thus to test alternative
hypotheses. Nevertheless, extra mortality is still an important construct
because (a) it helps to design experimental management actions that address its
potential causes; and (b) it is needed to simulate the range of effects of
alternative experimental actions to assess their relative risks and benefits.

In the PATH modeling framework, the ratio of these two values is known as
“D”. Like extra mortality, D cannot be directly measured, but must
be inferred from other measured quantities (e.g., Transport:Control ratios and in-river survival estimates from transportation studies for spring/summer chinook). Differences in the assumptions used to estimate D led to alternative hypotheses about both historical and future D values, for both spring/summer and fall chinook.

IMPLICATIONS FOR SELECTING A LONG-TERM MANAGEMENT ACTION

In general, the ability of transportation to recover stocks depends directly on D (i.e., more likely to recover stocks when D is high, less likely when D is
low). Drawdown actions were forecast to recover stocks over a wider range of D
values (Marmorek et al. 1998b, Peters et al. 1999, in press, Peters and Marmorek in press). The ability of both drawdown and transportation to recover stocks also depends on the extra mortality hypothesis – both actions were more likely to recover stocks
with the hydro hypothesis than with the regime shift or stock viability hypotheses.

Reducing these key uncertainties can help to identify the long-term management action that is best able to recover the stocks. There is an interaction between extra
mortality hypotheses and the D value: forecasts of recovery are generally more
sensitive to the extra mortality hypotheses if D is a high value. If D is high,
fewer transported fish die below Bonneville Dam. Other factors causing extra
mortality of all fish are then required to explain historical declines in
overall survival. If D is low, post-Bonneville mortality of transported fish
is sufficient to explain most of the observed historical declines in overall
survival rates, and extra mortality factors affecting all fish become less
important. This suggests that we should not measure D without also narrowing
down the extra mortality hypotheses, and vice versa.

APPENDIX 4Description of Weight of Evidence Process

This appendix provides more detail on the Weight of Evidence Process developed for spring/summer chinook (Fig. 8). We did not have time to initiate an analogous approach for fall chinook, steelhead, or sockeye. We first narrowed the problem down by sensitivity analysis to find the key uncertainties that affected the choice of management action (see Appendix 3). These analysis indicated that only seven of 14 uncertainties had a significant effect on outcomes, and three of these seven were particularly critical. The next steps involved an iterative series of written submissions, workshops, and syntheses to examine the evidence for and against alternative hypotheses for the seven key uncertainties. There were 25 submissions from PATH scientists (about 350 pages),
which we synthesized into a 150-page document (Marmorek et al. 1998c). We used
four criteria to evaluate alternative hypotheses:

1) the clarity of the hypothesis (i.e., clear specification of stressors affecting survival, without confounding);
2) existence of a reasonable mechanism or set of mechanisms by which the hypothesis operates;
3) consistency with empirical evidence (i.e., Do stock survival indices and hypothesized stressors vary across space and time in a manner consistent with the hypothesis? How well do different hypotheses fit empirical data such as reach survivals and recruits per spawner?); and
4) validity of the method of projecting the hypothesis into the future (i.e., are mathematical methods consistent with hypotheses and mechanisms that they were meant to represent? Are projections under current operations reasonable given recent
measurements not used in model calibration?).

Arguments and counter-arguments under each of these criteria were laid out systematically, with reference to supporting evidence in the submissions and other literature. All of the documentation (i.e., submissions and synthesis document) was provided to the Scientific Review Panel (SRP) for review. Three weeks later, the four SRP members attended a workshop in Vancouver at which experts in elicitation (people not involved in the PATH process) led them through the following steps (Peters et al. 1998):

1) training regarding process and judgmental biases;
2) clear definition of the judgments to be obtained, and explicit exclusion of any recommendations of specific management actions;
3) independent elicitation of the relative probability of alternative hypotheses, and the rationale for each SRP member’s conclusions;
4) aggregation and discussion of differences among experts; and
5) documentation.

Step 4 led the SRP members to recommend that PATH explore strong management experiments as an alternative approach to determining the probability of alternative hypotheses.

Subsequent to the workshop, the facilitation team applied the individual judgements of SRP members to the hypotheses in the decision analysis, calculated weighted-average outcomes, and compared those weighted averages to the situation in which all hypotheses were weighted equally (Peters et al. 1998). We found that applying the weights did not change the relative ranking of actions (A3 generally performed best), and had only a small effect on weighted average results, primarily because all four of the SRP members assigned similar weights to the passage/transportation models (which had large effects on the outcomes). These results were presented to the Implementation Team and a large contingent of interested public and media at a meeting in October 1998.

Experimental management requires spatial/temporal contrasts in treatments. In our initial scoping of experimental management actions (Marmorek et al. 1999), we determined that for many actions affecting the mainstem migration corridor (e.g., changes
in flow, methods or amount of smolt transportation, magnitude of hatchery
releases, harvest rates), one can only create temporal contrasts, not true
spatial contrasts (i.e., an independent control system). Partial spatial
contrasts are possible by comparing the performance of Snake River stocks with
stocks in the Lower or Mid-Columbia region (Fig. 1), using the approach developed by Deriso et al. (1996). However some of the Snake River treatments
could also affect Lower Columbia stocks, confounding the ‘control’
stocks. For example, upstream changes in flow, transportation, or hatchery
releases could all affect conditions for Lower Columbia River stocks during the
critical period of entry into the estuary and ocean. Because the four Snake River
projects are not used as storage reservoirs, drawdown actions are less likely
to have confounding impacts on downriver stocks. PIT-tags offer an opportunity
to create control groups within the same year for different transportation
methods, but the total number of tags required depends exactly on what one
defines as a control (e.g., smolts passing down river and never undetected at
any project, or simply smolts that are never put in a barge or truck), as well
as the level of marine survival of both treatment and control groups.

In our more detailed analyses of the five priority experimental management
actions, we used relatively simple models to simulate “true” future
survival changes associated with the candidate actions. We then assessed the
ability to learn by seeing how well the experiment and future monitoring could
estimate the “true” survival change in a fluctuating environment
(Peters et al. 2000; C. M. Paulsen and R. A. Hinrichsen, unpublished data).
The main metrics of how much can be learned from an action were expressed
in terms of the probability of estimating effects of an action over various
time frames, or, conversely, how long it would take to estimate an effect
with a certain level of confidence. Various criteria can be applied to determine
how long an experiment needs to be run to estimate effect sizes that reflect
the risk preferences of decision makers. The models also estimated various
conservation metrics for each action, which were compared to the ability to
learn to assess the trade-off between learning and conservation objectives.
To avoid the confounding of gradual changes in climate or ocean conditions,
we found that the most effective temporal contrast is switching an action
on and off in alternate years. Sub-basin actions such as carcass fertilization
could utilize more efficient experimental designs with both spatial and temporal
contrasts, and hence obtain a faster rate of learning. However, their survival
benefits are expected to be only modest, and insufficient on their own to
recover the stocks.

APPENDIX 6
Limitations of PATH's Models

PATH's escapement projections under the current operations scenario (option A1) were significantly higher that what was actually observed for recent years. This is
primarily because the models were designed to make relative comparisons of
actions over 100 years, and not short-term predictions. For example, year
effects were sampled from the last 40 years of stock performance, not only from
the poorer conditions that have generally prevailed since 1977. Although we have
no reason to suspect differences in the relative performance of actions from
this bias, the forecasted probabilities of survival and recovery are probably
over-optimistic for all actions. Recent model projections have confirmed that
most stocks would be likely to go extinct under current management if the poor
post-1985 year effects were assumed to continue into the future.

As with any model, certain processes were deliberately excluded because of both a lack
of understanding and a need to keep the scale of complexity reasonable. The
PATH life cycle models did not consider interactions between populations (e.g.,
straying) that could affect productivity through changes in genetic diversity.
The impacts of upstream storage projects were only considered in the models in
terms of changes in smolt travel time to the estuary; other mechanisms of
impact (e.g., changes in salinity) could also be important. The models also did
not consider changes in the quantity and quality of mainstem river habitat
other than those caused by the hydrosystem.

At a higher level, the PATH decision analysis only considered biological impacts.
A logical next step would have been to combine biological, social, and economic
performance measures into an integrated decision analysis that could examine
all uncertainties and trade-offs. However, the social, economic, tribal,
cultural, and recreation impacts of the actions under consideration were
examined in a separate effort from PATH (the Lower Snake River Feasibility
Study, under the direction of the U.S. Army Corps of Engineers), and it proved
to be infeasible to complete an integrated decision analysis. The Water Use
Planning Process developed to evaluate hydrosystem operations at 20 facilities
in British Columbia offers a promising approach to integrating social,
economic, and environmental objectives (McDaniels et al. 1999).

National Marine Fisheries Service (NMFS). 1995. Reinitiation of consultation on 1994-1998 operation of the Federal Columbia River Power System and Juvenile Transportation in 1995 and Future years. Northwest Region, National Marine Fisheries Service, Seattle, Washington, USA.

National Marine Fisheries Service (NMFS). 2000. Biological opinion. Reinitiation
of consultation on operation of the federal Columbia River Power System, Including the Juvenile Fish Transportation System, and 19 Bureau of Reclamation Projects in the Columbia River. Northwest Region, 21 December 2000. National Marine Fisheries Service web site: http://www.nwr.noaa.gov/1hydrop/hydroweb/docs/Final/2000Biop.html.