Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

4
Department of Homeland Security Decision
Requirements for Risk Management
With finite resources for biodefense, the United States must decide how to invest optimally to best
mitigate bioterrorism risk. Reducing uncertainty in risk analysis results has no direct impact on risk
reduction; only the implementation of effective risk management strategies can reduce risk.
âDepartment of Homeland Security, Bioterrorism Risk Assessment, 2006
RISK MANAGEMENT REQUIRES TIMELY, How much lead time (i.e., response time) is needed
ACCURATE INFORMATION to implement effective interventions? How much lead
time will surveillance provide? What assets should be
Those who, for reasons enumerated in previous chapters,
pre-positioned where and when to reduce and or man-
have need of information from the Department of Homeland
age the risk? Are transportation and communication
Security (DHS) regarding the risk of terrorist acts need that
resources sufficient to handle the surge in usage?
information to be accurate, timely, and valid. In 2002, the
â¢ How many and what type of staff would be needed
General Accounting Office (now Government Account-
to prevent, respond to, contain, or manage the conse-
ability Office) described the need to acquire and use the
quences of a bioterrorist attack? How should clinics
following information about the risk of terrorist acts in order
be designed for optimal provision of services in a
to achieve homeland security goals and objectives: (1) Who
crisis (e.g., mass vaccination, drug dispensing, patient
will do what and for what reason? (2) How (in what form),
triage, flow, and care) to effectively minimize the con-
where, and when will they do it? and (3) What will they use
sequences of a biological terrorism attack?
in order to do it? (GAO, 2002). Additionally, for bioter-
rorism, stakeholders and decision makers need answers to
These questions lead to further questions. For example, what
several types of questions (Danzig, 2003; Fischhoff et al.,
specific interventions would be needed (i.e., vaccination for
2006; Whitworth, 2006):
smallpox versus antibiotics for anthrax)? How fast should
they be delivered? What staff and supplies will need pre-
â¢ What is the probability of a particular biological terror-
positioning? (Cooper, 2006).
ist threat and how imminent is an event? What would
In reviews of unsuccessful past attempts to prevent,
the consequences of the event be, characterized in
prepare for, or respond efficiently to terrorist attacks, the
terms of when, where, and in what populations?
inability to put the right information into the hands of key
â¢ Can real-time detection be achieved to determine if an
decision makers at the right time and in an understandable
event has already occurred, if it is part of a larger plan,
format has been identified as a major factor in those failures.
or if it is a false alarm (perhaps intentionally generated
Failure can result in increased suffering and fatalities through
by terrorist actions)? How many ill and/or affected
setting the wrong priorities and policies, in the underfunding
people would be expected for attacks by different
of important programs, and in poorly conceived plans (Aaby
agents? How fast would different infectious agents in
et al., 2006). Several barriers to the effective use of informa-
a bioterrorist attack spread? How fast would people die
tion in decision making for the prevention of, preparedness
from different agents? How sick would survivors be?
for, and response to bioterrorism have been identified (Ware
â¢ What are the most effective ways to manage that risk?
et al., 2002):
How effective and feasible are different strategies for
the prevention, containment, and reduction of conse-
â¢ There are many stakeholders with varying degrees of
quences of a bioterrorism event? Would public health
authority for making and implementing decisions (e.g.,
measures be able to limit the spread of the infection
the responsibilities for domestic bioterrorist attacks in
to a small proportion of the total population? What
the United States may involve more than 100 different
impact would the availability and delivery of effec-
government organizations).
tive interventions have on potential consequences?
34

DEPARTMENT OF HOMELAND SECURITY DECISION REQUIREMENTS FOR RISK MANAGEMENT 35
â¢ Authorities, roles, responsibilities, tasks, and indicators The BTRA provides information to many stakeholders.
of success frequently are unclear and poorly understood DHS identifies the primary customers for information from
or coordinated. the BTRA as follows:
â¢ High volumes of different types of data and information
(e.g., subjective judgments, objective observations, â¢ White House Homeland Security Council: relative risks
historical data, analytical data, probabilistic data, mod- and overall vulnerabilities;
eling, simulation results) from disparate sources are â¢ Department of Health and Human Services: medical
presented in nonstandard and often poorly understood countermeasures needs;
formats, flooding the system as crises are unfolding. â¢ Department of Homeland Security/Infrastructure Pro-
â¢ Significant organizational friction frequently ex- tection: relative risks of different attack scenarios;
ists among the producers, owners, stakeholders, and â¢ Department of Homeland Security/Office of Intelli-
consumers of information. Critical information is gence and Analysis: high-leverage intelligence needs;
frequently owned or held by public and/or private â¢ Department of Homeland Security/Science and Tech-
organizations, accompanied by a general reluctance to nology: high-leverage scientific gaps;
share information across these sectors. â¢ Departments of Agriculture and of Health and Human
â¢ The producers and owners of important information Services: food security;
often compartmentalize it in order to protect secu- â¢ Environmental Protection Agency: water security;
rity or to protect sources, at the expense of the timely and
and integrated sharing of data and interpretation of â¢ Department of Agriculture: agricultural agents and
information. protection of the food supply.
â¢ Decision makers often have widely varying objec-
tives and frequently little understanding of the medi- DHS stakeholders need risk analysis, including risk
cal and scientific background needed to inform their management, for strategic planning, operations, and foren-
decisions. sics. Further, Homeland Security Presidential Directive 10
(HSPD-10): Biodefense for the 21st Century (The White
House, 2004) states that the âUnited States requires a con-
THE BIOLOGICAL THREAT RISK ASSESSMENT
tinuous, formal process for conducting routine capabilities
SHOULD SUPPORT RISK MANAGEMENT
assessments to guide prioritization of our on-going invest-
The GAO (2002) report identified risk assessment as ments in biodefense-related research, development, plan-
an important tool and source of information for strategic ning, and preparedness. These assessments will be tailored
decision making for the prevention of bioterrorism, risk to meet the requirements in each of these areas. Second,
reduction, preparedness, and response to bioterrorism. As the United States requires a periodic senior-level policy
described in Chapter 3 of this report, the DHS Biological net assessment that evaluates progress in implementing
Threat Risk Assessment (BTRA) of 2006 is one of the first this policy, identifies continuing gaps or vulnerabilities in
terrorism risk assessment efforts to integrate information our biodefense posture, and makes recommendations for
from a variety of sources to meet information needs. Further, re-balancing and refining investments among the pillars
the BTRA of 2006 presents sensitivity analyses that permit of overall defense policy.â To the extent that the BTRA of
an examination of the impact of different measures that could 2006 (with its subsequent improvements and revisions) is
be taken to mitigate identified consequences of interest (i.e., used for risk analysis, the committee believes that it is most
morbidity and mortality). However, as noted in Bioterror- applicable to supporting strategic decisions (those that ad-
ism Risk Assessment, the DHS 2006 report that describes the dress the setting of priorities and policies, the acquisition
BTRA methodology, with finite resources for biodefense, the and pre-positioning and/or allocation of resources, and the
United States must decide how to invest optimally to best development of infrastructure), but that it is not designed to
mitigate bioterrorism risk. As that report points out, risk as- support operations or forensics.
sessment alone has no direct impact on risk reduction; âonly In 1997, the Presidential/Congressional Commission on
the implementation of effective risk management strategies Risk Assessment and Risk Management (1997 a,b; Omenn,
can reduce riskâ (DHS, 2006). 2003) agreed on a framework for environmental health risk
management, which is applicable to managing risks involved
Recommendation: Subsequent revision of the BTRA with bioterrorism. This framework has six stages: (1) formu-
should increase emphasis on risk management. An in- late the problem in a broad public health context, (2) analyze
creased focus on risk management will allow the BTRA to
better support the risk-informed decisions that homeland â
Rear Admiral Jay Cohen, Undersecretary of Science and Technology,
security stakeholders are required to make. Department of Homeland Security. 2007. âDHS Science and Technology:
Enabling Technology to Protect the Nation.â Briefing to the committee,
February 9, 2007, Washington, D.C.

36 DEPARTMENT OF HOMELAND SECURITY BIOTERRORISM RISK ASSESSMENT
the risks, (3) define the options to address the risks, (4) make that stakeholders have in the modelâs methods, the validity
sound risk reduction decisions, (5) implement those actions, of assumptions and data used to develop the model, and
and (6) later evaluate the effectiveness of the actions taken. the level of understanding of the modelâs outputs. Lack of
The BTRA of 2006 focused on risk assessment, which confidence can be caused by an insufficient understanding
addresses stages 1 and 2 above. However, DHS intends for its of or disagreement with relationships hypothesized among
BTRA to be used by risk managers to âtest and evaluate risk variables, the mathematical foundations of the model, and/
mitigation strategies and their impact on bioterrorism riskâ or the validity of assumptions and values assigned to the
(DHS, 2006, Ch. 1, p. 3). To broaden the focus to include modelâs parameters. All of these problems can be mitigated
risk management, stages 3 and 4 must be addressed. Stage by improved transparency.
3, defining the options to address the risks, includes identi-
fying potential countermeasures and estimating the costs of
RISK ASSESSMENT TRANSPARENCY
deploying the countermeasures. So that the risk assessments
IMPROVES CONFIDENCE
provided by the BTRA can be effectively used, each poten-
tial countermeasure must be mapped to a set of parameters In contemplating a complex problem involving uncer-
within the model. Stage 4, making sound risk reduction tainty and risk, such as that involved with the threat of bioter-
decisions, requires several related activities (Boardman et rorism, mental arithmetic can be riddled with error, while risk
al., 2006): assessment models enable repeatable calculations, including
sensitivity analyses, which explore the effects of uncertain
â¢ Estimation of risk reduction (in, for example, expected parameters on important consequences. However, risk as-
lives, life-years, or quality-adjusted life-years) obtained sessment models can fail to include important knowledge
by allocating countermeasures, as discussed in Chapter that is not readily quantified and/or understood, potentially
2 of this report; compromising the validity of model outputs (Fischhoff et
â¢ Optimization of the allocations, which identifies, for al., 2006).
each given resource level or budget, the allocation of The accuracy of quantitative bioterrorism risk assess-
countermeasures that maximizes total risk reduction ment models and the confidence placed in them depend on
(or equivalently, for a given level of risk reduction, the validity of the assumptions and the availability of sound
identifies the least-cost deployment of countermeasures data for each of the biological agents being analyzed. A
for achieving a particular level of risk), as discussed in good model based on strong data, such as the one reported
Chapter 7 of this report; in Whitworth (2006) that describes the difference between a
â¢ Optimization of risk-benefit, which, given the optimal response for an anthrax attack and that for a smallpox attack,
allocation of resources for different budget levels and can inform judgment about the effectiveness of different in-
a willingness-to-pay value for incremental risk reduc- terventions (i.e., antibiotics for anthrax versus vaccinations
tion, identifies the best overall level of resources (and for smallpox) and the pre-positioning of staff and supplies
corresponding best allocation); and to respond to an attack effectively.
â¢ Valuation of options, or consideration of how the re- Conversely, the lack of data and/or uncertainty in model
sults from the previous stages are likely to change if parameters also can have important implications for the
additional countermeasures are added, and using this degree of confidence placed in the results of risk assess-
information, making decisions about which additional ment models (Elderd et al., 2006). Unfortunately, data for
countermeasures are most worth developing. key parameters of many biothreat agents of concern are
not available. If decision makers understand and trust
the model, they will be more likely to use it with differ-
TRANSPARENCY OF RISK ASSESSMENT IS
ent assumptions and to test different response strategies
NECESSARY FOR SUCCESSFUL RISK MANAGEMENT
(
Â­ Fischhoff et al., 2006).
Transparency, as described by Oliver (2004), has been While transparency is a major factor in establishing
defined in different dictionaries as âfree from guile,â âcandid confidence and trust in the methods of and outputs from
or open,â or âforthright,â and has been applied to business risk assessment models, modeling is an important step to-
and organizations as âallowing others to see the truth without ward transparency as it requires that assumptions be made
trying to hide or shade the meaning or altering the facts to explicit. However, achieving transparency also requires the
put things in a better light.â Oliver summarizes the current careful, explicit documentation of a modelâs mathematical
use of the word transparency as âletting the truth be available and structural foundations and of the sources of data used
for others to see if they so choose, or perhaps think to look, in the analysisâa prerequisite for any scientific studyâfor
or have the time, means, and skills to look,â and involving
â
Marc Lipsitch, Harvard School of Public Health. âNotes to the National
âactive disclosure.â
Whether and how often risk assessment models are used Research Council Committee on Methodological Improvements to the
DHSâs Biological Agent Risk Analysis.â Written communication to the
in risk management will depend on the level of confidence committee, February 2007.

DEPARTMENT OF HOMELAND SECURITY DECISION REQUIREMENTS FOR RISK MANAGEMENT 37
the purpose of facilitating external review. It is essential that THE DEPARTMENT OF HOMELAND SECURITYâS
analysts document the following: (1) how they construct BTRA OF 2006 WAS NOT TRANSPARENT
risk assessment models, (2) what assumptions are made to
As described in Chapter 3 of this report, the model used
characterize relationships among variables and parameters
in the BTRA of 2006 is extremely complex, with 17 stages
and the justifications for these, (3) the mathematical foun-
and thousands of parameters for each of 28 biothreat agents
dations of the analysis, (4) the source of values assigned to
of concern (DHS, 2006). The considerable data that are
parameters for which there are no available data, and (5)
lacking for many of the parameters and probabilities in
the anticipated impact of uncertainty for assumptions and
the model may lead to questions about the validity of the
parameters (Brisson and Edmunds, 2006).
modelâs output and to a lack of confidence and trust in the
When working with classified or sensitive information, as
results. Moreover, the results of simulations are presented in
is the case with bioterrorism prevention, preparedness, and
graphs, charts, and tables that are also complex and difficult
response, there may be need to restrict the access to some
to interpret and use.
information to certain groups of users to protect overall
The committee also finds the documentation for the model
security. However, security or confidentiality concerns that
used in the BTRA of 2006 to be incomplete, uneven, and
can negatively affect the level of transparency reached in risk
extremely difficult to understand. The BTRA of 2006 was
assessment modeling include the following:
done in a short time frame. However, deficiencies in docu-
mentation, in addition to missing data for key parameters,
â¢ The compartmentalization of model development,
would make reproducing the results of the model impossible
algorithms, and execution for security concerns and to
for independent scientific analysis. For example, although
protect information and data sources;
Latin Hypercube Sampling is mentioned in the description
â¢ Private-sector reluctance to share information, com-
of the model many times as a key feature, no actual sample
monly due to the protection of proprietary consider-
design is specified. Although antithetic sampling methods
ations, across sectors; and
(e.g., matched samples or reused random number streams)
â¢ The need to balance civil liberties of citizens against the
evidently are employed, insufficient details are provided
need to keep important classified and sensitive informa-
on how or where these numbers are generated, precluding
tion out of the hands of terrorists.
a third party, with suitable software and expertise, from
reproducing the resultsâviolating a basic principle of the
Given the importance of establishing the confidence of de-
scientific method.
cision makers and stakeholders in risk assessment models,
Finally, the current BTRA implementation must be run
it is essential to strive for the highest level of transparency
on a custom computer cluster in a DHS contractor facility
possible while being sensitive to the need to restrict access
taking many hours to compute; also the data are cumber-
to those with a need to know.
some to prepare. This makes answering âwhat-ifâ questions
of the type that stakeholders are likely to ask an expensive
THERE ARE SEVERAL OTHER WAYS and slow process.
TO BUILD CONFIDENCE During the course of this study, in response to technical
questions posed by three members of the committee, DHS
The confidence of decision makers in the information
provided the committee with a technical document (DHS,
generated by a risk assessment model can be increased in
2007) that includes answers to the questions posed and which
several ways, and increased confidence will heighten the
became an essential piece of documentation missing from the
likelihood that the information will be used. Decision makers
original publications provided to the committee. In addition,
can be engaged iteratively during model development; this
the committee asked another expert to make an independent
is critical in order to increase their understanding of how the
review of the methodology employed for the model used in
model is being constructed and to ensure that their informa-
the BTRA of 2006 using the originally published material
tion needs will be met. Complex systems can be simplified
and the technical addendum. The independent review is
to more-readily-understood scenarios and coupled with the
reproduced in this report as Appendix I. The author of that
capability of conducting real-time sensitivity analyses. In
review encountered difficulties similar to those described
addition, experts can conduct independent, periodic external
here; one of three suggestions in Appendix I is âto report
reviews; doing so is critical in order to assure stakeholders
future results in a scientific fashion than can be reviewed by
that the appropriate inputs and models are being used for risk
scientistsââa suggestion that is echoed in the next recom-
assessment. If the risk assessment model and/or its assump-
mendation of this committee (see below).
tions are not accurate or appropriate, the results of a model
Some of the probabilities of important consequences in
and accompanying sensitivity analyses can give a false sense
the model used in the BTRA of 2006 were extremely close
of security in the results, may lead to inappropriate policy
decisions (Brisson and Edmunds, 2006), and ultimately will â
Alan R. Washburn, Distinguished Professor Emeritus of Operations
lead to a lack of confidence in the use of these models. Research, Naval Postgraduate School, Monterey, California.

38 DEPARTMENT OF HOMELAND SECURITY BIOTERRORISM RISK ASSESSMENT
to zero. When risk is expressed in numerical form, whether THE BTRA SHOULD BECOME A
or not decision makers are motivated to take action will DECISION SUPPORT SYSTEM
frequently depend on how confident they are in the number.
Decision support systems (DSSs) are interactive informa-
Human beings are known to have difficulty in rationally
tion technology platforms that facilitate the use of informa-
processing numbers and probabilities (Paulos, 1988; Tversky
tion in complex decision making. The goals of DSSs are to
and Kahneman, 1973, 1974), and when probabilities are ex-
improve the efficiency with which users make decisions and
tremely small, decision makers will often give them greater
to improve the effectiveness of their decisions (Shim et al.,
weight than is appropriate or possibly ignore them altogether,
2002; Pearson and Shim, 1995). DSSs are especially helpful
at great peril when there are potentially significant and large
in decision-making situations where there are multiple deci-
consequences. Thus, extreme caution is needed to avoid an
sion makers with different roles, functions, and responsibili-
under- or overinterpretation of results that may cause errors
ties, and different types and sources of data and databases.
in decision making when probabilities of consequences are
There are many different designs for DSSs, but in gen-
estimated to be near zero, such as is the case with the model
eral they include the following components: (1) database
used in the BTRA of 2006. Systematic use of well-grounded
management capabilities that provide access to relational
models can guide decision makers to account for these prob-
databases, information, and knowledge from a variety of
abilities correctly.
sources; (2) modeling and modeling management functions;
and (3) a simple user-interface component that supports in-
Recommendation: DHS should maintain a high level of
teractive queries, reporting, and graphic functions (Marakas,
transparency in risk assessment models, including a com-
1999; Druzdzel and Flynn, 2002; Shim et al., 2002; Ware
prehensive, clear mathematical document and a complete
et al., 2002). DSSs have been developed to support specific
description of the sources of all input data. The documen-
decisions involved with bioterrorism prevention, prepared-
tation should be sufficient for scientific peer review.
ness, and response (Bravata et al., 2002, 2004; Ware et
al., 2002). Bravata et al. (2002) identified 217 information
To carry out this recommendation, DHS should do the
technology DSSs that were of potential use to clinicians and
following:
public health officials in the event of a bioterrorism event.
One example of a DSS that facilitates data-based decision
â¢ Solicit input from multiple stakeholders involved with
making for resource allocation problems and emergency
the prevention of bioterrorism (whether directed at
response planning is a stand-alone, large-scale DSS, called
humans or at agriculture), preparedness, and response
RealOpt. RealOpt pairs a flexible simulation component with
throughout the development of the model in order to
a set of analytical, decision-making algorithms. The system
enhance their understanding of and therefore their con-
comprises three integrating components:
fidence in the model, its data inputs, and its results;
â¢ Clearly state the objectives and carefully define the
â¢ A simulation manager that runs simulations with
input variables, sources of data, and associated conse-
c
Â­ hanges in input parameters in order to investigate
quence models; make assumptions explicit; and justify
behavior and bottlenecks in the system for different sce-
the values that are assigned to variables, parameters,
narios, and calculates various outcome statistics (e.g.,
and probabilities;
average wait time, queue length, and utilization rates);
â¢ Provide a guide to facilitate the interpretation of results,
â¢ An optimization manager that stores algorithms and
especially in the context of important outcomes that are
fast heuristics and iteratively calls on the simulation
estimated to occur with probabilities approaching zero;
manager to resolve and update resource-allocation
and
statistics (e.g., to maximize throughput, to minimize
â¢ Conduct a scientific, periodic external review of the
staff usage to satisfy a specified throughput); and
validity of the risk assessment modelâs development
â¢ A user-interface manager and linker module that con-
and analysis; carefully and completely document how
nects the input of data to a display of results, including
the model is developed and its mathematical founda-
a graphics algorithm that allows users to design specific
tions, using terms from a widely accepted, standard
floorplans of different patient care and dispensing fa-
technical lexicon in understandable language, such that
cilities for vaccinations and different medications (Lee
an independent, external panel of experts can duplicate
et al., 2006).
the results; have an independent blue team perform
complete scenario dissection for selected paths through
The characteristics of DSSs that are effective in giving de-
the entire event tree; and take care to allow the widest
cision makers access to the understandable information that
possible review subject to security requirements.
they need and can use for decision making are as follows:

DEPARTMENT OF HOMELAND SECURITY DECISION REQUIREMENTS FOR RISK MANAGEMENT 39
â¢ The DSS clearly states its objectives and desired out- with extremely small probabilities. Scenarios are especially
comes (i.e., timely, quality decisions); useful when decision makers are inexperienced in systems
â¢ It addresses consequence and identifies key questions thinking. They may help inexperienced systems thinkers
of stakeholders who were involved in framing the avoid using unrealistic assumptions, which can lead to the
problems; development of incomplete, infeasible, or ineffective plans
â¢ It is user-friendly for a variety of stakeholders and does (Whitworth, 2006).
not require sophisticated information technology skills Danzig (2003) described a number of situations in which
for its operation; planning scenarios could be used. First, they can bring
â¢ The DSS is flexible, efficient, and includes an easy-to- awareness to sets of specific circumstances and hypothesized
access help desk and documentation; chains of events, which, if understandable and conveyed
â¢ It is portable across different computer platforms and in a compelling manner such that the decision maker has
personal digital assistants; confidence in the method, will have a greater likelihood for
â¢ It provides results that are well matched to decision resulting in action. Second, they can help in the development
objectives. Decision makers can ask for and easily get of coordinated actions and plans among multiple stakehold-
results of new simulations reflecting different assump- ers by keeping everyone focused on the same narrative or
tions on how an event will present alternative responses alternative. Third, a planning scenario can serve as a refer-
and interventions leading to different outcomes; ence case against which alternative strategies can be com-
â¢ The DSS requires minimal computation time for simu- pared and tested. Fourth, planning scenarios can be used to
lationâseconds or minutes versus hours for individual establish resource and other requirements needed to prevent
simulation runs; or respond to potential events.
â¢ It provides accurate results and information that can be Critics of scenarios are concerned that their use may make
used to gauge how much confidence can be placed in assumptions unclear or inexplicit, complicating external
model outputs. Systematic checks of data quality are review and assessment and making their validity difficult to
built in to the analysis system and display of results; assess. When insufficient detail is provided, different stake-
â¢ It has displays that are simply designed with high- holders may arrive at different perceptions of a scenario and
resolution data; it has relevant information presented end up coming to incompatible conclusions or developing
and conveyed in an understandable way and accessed uncoordinated or incompatible plans. For this reason, scenar-
easily; and its tables and graphs are well labeled. The ios must reflect the most complete, explicit, and transparent
DSSâs displays should be accompanied by annotation, details available and allow for a ready comparison of per-
details, other supplements, including limitations, aids ceptions among the various stakeholders. Finally, scenarios
to interpretation of risk, confidence limits around risk, can become too rigid. They require continual updating as
and confidence in solutions. new information becomes available. However, effectively
planning for these possibilities can mitigate these organiza-
Recommendation: Subsequent revision of the BTRA tion problems. The committee recognizes the difficulty in
should enable a decision support system that can be run preparing and validating accurate and useful scenarios. For
quickly to test the implications of new assumptions and that reason, it suggests that, as with other BTRA documenta-
new data and provide insights to decision makers and tion, any such scenarios be peer reviewed.
stakeholders to support risk-informed decision making.
Sensitivity Analysis Is Important for Validation
Use Scenarios
Sensitivity analysis has been defined as the determination
A successful DSS, as described above, would facilitate the of how âuncertainty in the output of a model (numerical or
use of scenarios. Mathematical models (e.g., risk assessment otherwise) can be apportioned to different sources of uncer-
models) often are so complex that their results are not easily tainty in the model inputâ (Saltelli and Tarantola, 2002). The
understood, met with confidence, and used. Decision makers purposes of sensitivity analyses are to (1) give users of risk
commonly deal with the uncertainty of future events by using assessment models information that they can use to identify
âwhat-ifâ scenarios, which can bound uncertainty and bring key parameters and explore a range of impacts that can be
multiple stakeholders together to consider a shared, selected expected with changes in input and parameter values, and
set of hypothesized chains of events in narrative form, and to to evaluate the confidence they can place on model outputs;
consider alternatives (Pomerol, 2001). Scenarios can make (2) identify sources of uncertainty in the model when as-
abstract or nebulous threats more concrete, which can help sumptions and parameters vary across possible scenarios;
decision makers avoid becoming lost in trying to assimilate (3) aid planners in comparing alternative strategies and test
large numbers of variables, relationships, and parameters how a given plan would work should assumptions be wrong;

40 DEPARTMENT OF HOMELAND SECURITY BIOTERRORISM RISK ASSESSMENT
(4) help decision makers make the best possible decisions strategies for tools that are easier for users to employ are
in the presence of uncertainty; and (5) set priorities for the presented in the next subsection.
collection of additional information (Meltzer, 2001; Whit- The purpose of the sensitivity analysis will affect the
worth, 2006). method to use for the analysis. Borgonovo (2006) described
Because considerable data are lacking for many of the three families of analytic techniques (i.e., variance-based,
parameters and probabilities in the model used for the BTRA input-output correlation, and moment-independent analy-
of 2006, there may be an accompanying lack of confidence in ses), the choice of which would depend on the stated purpose
the results. Thus, being able to conduct a sensitivity analysis of the sensitivity analysis.
is an essential feature of the BTRA model if it is to be used Precautions must be taken when drawing conclusions
for decision making. Although the BTRA was apparently from sensitivity analyses, as the accuracy of the informa-
developed so that the impact of different parameters on con- tion is conditional on the validity of the underlying model
sequences of importance could be assessed through sensitiv- structure and the methods used to exercise it. If the structure
ity analysis, the process for running sensitivity analyses cur- of the risk assessment model and/or the assumptions used in
rently is not interactive and appears quite âuser-unfriendlyâ the model are not accurate or appropriate, then the results of
and cumbersome. To see results of the model for different a sensitivity analysis can give a false sense of security in the
scenarios (changing values for parameters and in different results and may lead to inappropriate policy decisions (Bris-
branches of the tree for different agents), changes in values son and Edmunds, 2006). Extreme caution also is needed to
must be submitted to analysts who rerun the model. Results avoid a misinterpretation or overinterpretation of results and
are then evidently available hours to days later, making the the making of errors in decision making when valid data for
sensitivity analysis process difficult for decision makers parameters or probabilities are lacking.
to use immediately. Finally, the results of simulations are
presented in graphs, charts, and tables that are complex and
Create a Context for Use
difficult to interpret and use.
For these reasons, the committee questions whether the In addition to the approaches discussed so far, strengthen-
results of the 2006 Biological Threat Risk Assessment model ing the overall environment for data-based decision making
are answering the highest-priority questions of different deci- is critical. A comprehensive and continually updated set of
sion makers; whether they are being conveyed in the most guidelines, protocols, and checklists that provide essential
understandable, useful, and compelling manner possible; and details on clear courses of actions that decision makers would
whether the current sensitivity analysis feature is meeting make, conditional on the information made available to them
information needs. User-friendly sensitivity analysis could from risk assessment models analyzing a set of structured
also be a part of any DSS. scenarios, must be developed, tested, and in place. These
The uncertainty or lack of data and/or errors in measure- materials should be prepared for different stakeholders and
ment for many key variables and parameters in risk assess- should include a range of possible decisions and actions, by
ment models for potential bioterrorism events can affect scenario, for different authorities, roles and responsibilities,
the confidence that decision makers place on the output of desired outcomes, and benchmarks for tracking progress for
the model. Accepted good modeling practices require that different scenarios.
models be continually tested and validated by evaluating the Different strategies and protocols should be practiced as
effect of uncertainties with regard to values of parameters tabletop and TOPOFF exercises to identify areas where ad-
and probabilities of the model. Sensitivity analysis has be- ditional attention, planning, resource acquisition and alloca-
come an accepted and important approach to the testing and tion, and practice are required. A trained workforce within
validation of risk assessment models of complex systems and across multiple sectors, agencies, and institutions in the
(Borgonovo, 2006). public and private sectors is essential.
In the future, it will be important to move the sensitivity Relevant, accurate, timely information that is available in
analysis from questions about risk assessment (for example, understandable formats and terms, with guides to the inter-
How does uncertainty about the infectious dose for this agent pretation of outcomes, is critical. Environments that support
change my expected consequences?) to questions about risk the use of data in decision making are those in which the right
management (for example, If I had improved knowledge resources (e.g., staff, drugs, vaccines, respirators, other) are
about the infectious dose for this agent, would I adopt dif- pre-positioned strategically and available to the right staff
ferent countermeasure strategies?). Currently, simulation at the right time.
runs take an extended period of time to run owing to the
complexity and size of the model and its input data; outputs
from the model are not presented in easily used, interactive,
understandable and compelling formats. Strategies for reduc- â
In
Chapter 7, the committee discusses the benefits of red teaming in
ing the complexity of the model are presented in Chapter 7; TOPOFF (Top Officials) exercises.

The mission of Department of Homeland Security Bioterrorism Risk Assessment: A Call for Change, the book published in December 2008, is to independently and scientifically review the methodology that led to the 2006 Department of Homeland Security report, Bioterrorism Risk Assessment (BTRA) and provide a foundation for future updates.

This book identifies a number of fundamental concerns with the BTRA of 2006, ranging from mathematical and statistical mistakes that have corrupted results, to unnecessarily complicated probability models and models with fidelity far exceeding existing data, to more basic questions about how terrorist behavior should be modeled.

Rather than merely criticizing what was done in the BTRA of 2006, this new NRC book consults outside experts and collects a number of proposed alternatives that could improve DHS's ability to assess potential terrorist behavior as a key element of risk-informed decision making, and it explains these alternatives in the specific context of the BTRA and the bioterrorism threat.

Welcome to OpenBook!

You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.