WPS6765
Policy Research Working Paper 6765
Making Informed Investment Decisions
in an Uncertain World
A Short Demonstration
Laura Bonzanigo
Nidhi Kalra
The World Bank
Sustainable Development Network
Office of the Chief Economist
February 2014
Policy Research Working Paper 6765
Abstract
Governments invest billions of dollars annually in long- energy security. The analysis shows that Robust Decision
term projects. Yet deep uncertainties pose formidable Making can help decision makers answer specific and
challenges to making near-term decisions that make long- useful questions: How do options perform across a wide
term sense. Methods that identify robust decisions have range of potential future conditions? Under what specific
been recommended for investment lending but are not conditions does the leading option fail to meet decision
widely used. This paper seeks to help bridge this gap and, makers’ goals? Are those conditions sufficiently likely that
with a demonstration, motivate and equip analysts better decision makers should choose a different option? Such
to manage uncertainty in investment decisions. The paper knowledge informs rather than replaces decision makers’
first reviews the economic analysis of ten World Bank deliberations. It can help them systematically, rigorously,
projects. It finds that analysts seek to manage uncertainty and transparently compare their options and select one
but use traditional approaches that do not evaluate that is robust. Moreover, the paper demonstrates that
options over the full range of possible futures. Second, analysts can use the same data and models for Robust
the paper applies a different approach, Robust Decision Decision Making as are typically used in economic
Making, to the economic analysis of a 2006 World Bank analyses. Finally, the paper discusses the challenges in
project, the Electricity Generation Rehabilitation and applying such methods and how they can be overcome.
Restructuring Project, which sought to improve Turkey’s
This paper is a product of the Office of the Chief Economist, Sustainable Development Network.. It is part of a larger
effort by the World Bank to provide open access to its research and make a contribution to development policy discussions
around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors
may be contacted at lbonzanigo@worldbank.org .
The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development
issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the
names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those
of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and
its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent.
Produced by the Research Support Team
Making Informed Investment Decisions in an Uncertain
World: A Short Demonstration
Laura Bonzanigo 1 and Nidhi Kalra 2
JEL: C44, D81, H54, O21, O22
Keywords: decision making under uncertainty, long-lived infrastructure investments,
economic analysis, robust decision making
1
Sustainable Development Network, Office of the Chief Economist, The World Bank; Department of
Economics, Ca’ Foscari University, Venice
2
Sustainable Development Network, Office of the Chief Economist, The World Bank
1 Introduction
Each year governments invest billions of dollars towards development. Many of these
investments – in energy, land use, transportation, and other sectors – are long-term and will
shape the course of development in their countries. Yet we live in an unpredictable world
governed by competing beliefs and preferences. Our decisions are engulfed in deep
uncertainty about the long-term cost of energy inputs, the impact of climate change, and a
host of other factors that shape our decisions. Deep uncertainty occurs when the parties to a
decision do not know—or do not agree on— the likelihood of future events, the best model
for relating actions to outcomes, or the value of potential outcomes (Lempert et al., 2003).
Deep uncertainties pose formidable challenges to making near-term decisions that make
long-term sense. Governments grapple with deep uncertainties daily. Traditionally, they
have asked, “Which investment option best meets our goals given our beliefs about the
future?” Such approaches, sometimes called “Predict-then-Act”, hinge on our accurately
predicting and then reaching consensus on what the future will bring (Hallegatte et al., 2012;
Lempert and Kalra, 2011). But disagreements about the future can lead to gridlock. Worse,
investments tailored to one set of assumptions about a deeply uncertain future often prove
inadequate or even harmful if another future comes to pass. Governments and international
development organizations increasingly recognize that current approaches to decision
making struggle to meet these ubiquitous challenges (Rennkamp, 2012; Ranger, 2013; IEG,
2010, 2012; The World Bank, 2013).
For instance, throughout the World Bank, there is a growing recognition that deep
uncertainties need to be better managed in decision making (Hallegatte et al., 2012; IEG,
2010; Independent Evaluation Group (IEG), 2012). In 2010, The World Bank’s Independent
Evaluation Group released a report pressing World Bank staff to conduct more rigorous
economic analyses of investment options (IEG, 2010) and to use these analyses to inform
decisions. This was endorsed by the World Bank’s Board of Directors in a Guidance Note to
project managers (The World Bank, 2013). More recently, a more explicit consideration of
risk and uncertainty is also one of the key messages of the World Development Report 2014
(The World Bank, 2014).
Many methods have been developed over the last half-century to help decision makers
manage deep uncertainties and make investments that are robust to the unpredictable future.
These approaches ask, “Which investment option best meets our goals given that we cannot
know what the future will bring?” (Hallegatte et al., 2012). These methods seek to identify
robust decisions – those that satisfy decision makers’ objectives in many plausible futures,
rather than being optimal in any single best estimate of the future (Lempert et al., 2013).
They have been recommended for investment lending but are not widely used in practice,
leaving many decisions vulnerable to surprise. One reason may be that we do not
understand well the extent to which deep uncertainties affect lending decisions, or know the
usefulness and practicality of these methods for investment lending.
We seek to help bridge this gap. In this study, first we review the economic analysis of ten
World Bank projects approved between 2002 and 2011 in order to understand how they
managed risk and uncertainty. Then, we test and demonstrate the practicality and value-
added of new methods for long-term infrastructural investment lending decisions. Research
suggests that having practical tools to solve a problem can increase one’s awareness of the
2
problem and motivation to solve it (Coombes and Devine, 2010; Kolb, 1984). Our hope is
that a straightforward demonstration of these methods may motivate and equip analysts to
better manage uncertainty in lending decisions.
In particular, we apply Robust Decision Making (RDM, Lempert et al., 2003) to the economic
analysis of a prior World Bank project, the Electricity Generation Rehabilitation and
Restructuring Project, which in 2006 sought to improve Turkey’s energy security in part by
increasing near-term energy supply. We use the same data and models utilized in the
original analysis, but in a different way. Rather than seeking to inform electricity investments
in Turkey with predictions of the future, we seek to inform them with assessments of their
robustness to an unpredictable future.
The original decision was to rehabilitate an existing coal plant. Other options included
building new coal-fired, gas-fired, or other power plants. Decision makers were concerned
with two key metrics: a) whether the investment passed a cost minimization metric – i.e.,
produced electricity at lower cost than any other option, and b) whether the investment
passed a cost-benefit test, in this case had a rate of return of at least 12%. We evaluated
each option according to these metrics in 500 plausible future states of the world that varied
under seven different sources of uncertainty.
We used the results to answer a series of specific and useful questions:
• How do decision makers’ options perform across a wide range of potential future
conditions?
• Under what specific conditions does the leading option fail to meet decision
makers’ goals?
• Are those conditions sufficiently likely that decision makers should choose a
different option that is more robust?
Our analysis suggests that methods like RDM can provide decision makers with much more
salient information about the merits and vulnerabilities of different options. This can focus
decision makers’ attention on the uncertainties that matter most to a decision. It can make
them aware of the important trade-offs and of the actions they could take to reduce their
vulnerability. Ultimately, it puts the decision back in the hands of decision makers by helping
them take measured risks and be less vulnerable to surprise.
Importantly, the purpose of this exercise is not to prove the original project decision right or
wrong or to re-create the analysis under today’s conditions. At the time of this project, for
instance, climate change mitigation and the possibility of a price on carbon were not
widespread concerns, and this analysis reflects the original priorities of decision makers
related to energy security. The same analysis today would almost certainly include these
considerations. Rather, our aim is to demonstrate that an analysis of robustness can be
incorporated into projects’ standard economic analyses using the same data and models,
and to examine the different kinds of information that emerge from a Predict-then-Act versus
a robustness analysis. Moreover, while our analysis focuses on a World Bank case study,
we believe this methodology holds value for the broader international lending community.
We believe methods like RDM can be readily incorporated into cost-benefit and other
economic analyses that analysts perform every day, with few additional resources. We hope
3
this analysis will be a useful template by which projects can better manage uncertainty in
their investment decisions.
2 Review of World Bank Projects
The typical approach to economic analysis is to conduct a cost-benefit analysis of the
preferred investment option under best estimate predictions of the relevant future conditions.
Consistent with the World Bank’s manual on economic analyses, analysts sometimes
conduct a sensitivity analysis around the best estimate projection to address uncertainty
(Belli, 2001). Occasionally they may replace a single projection with a probabilistic projection,
assigning likelihoods to a variety of future outcomes.
These approaches have a common underpinning. Often termed Predict-then-Act (Dessai
and Wilby, 2010), they seek to characterize the future (i.e. make predictions) and then
measure policy options against the characterizations to determine the best near-term course
of action. Predict-then-Act forms the basis of several analytic methods commonly used to
evaluate investments (Lempert and Kalra, 2011; Hallegatte et al., 2012) – including
traditional risk analysis, cost-benefit analysis (Arrow and Fisher, 1974), and real options
analysis (Henry, 1974).
Yet the World Bank increasingly recognizes that such approaches may not help manage
deep uncertainties that arise in many lending decisions (Hallegatte et al., 2012; IEG, 2010;
Independent Evaluation Group (IEG), 2012). In this study, we sought to understand the
World Bank’s economic analyses in greater detail by reviewing the Project Appraisal
Documents (PADs) 3 of ten projects that were approved between 2002 and 2011. PADs are
compulsory documents that projects must to submit to the World Bank’s Executive Board for
lending approval. They contain basic information on the proposed projects, such as Country
Ministries involved, project length and its objectives, a section on key risks and plausible
mitigation measures, and the economic and financial analyses. The economic analysis
should quantify the benefits and the costs of the proposed project and of the other options
examined.
In our review of ten PADs we asked several questions:
• How many and which alternative options did the project originally evaluate?
• What uncertainties did analysts identify in the project narrative?
• Which of these were carried into the economic analysis of projects?
• How did the projects manage uncertainty?
While our small sample size lends no statistical weight to the answers, it enriches the
observations made by the Independent Evaluation Group and others at the World Bank. We
randomly selected investment projects in two sectors – energy and transportation – that
were characterized by the World Bank’s project database as addressing climate change, a
deep uncertainty.
3
PADs summarize the project team’s assessment of a lending project and are the basis for the World
Bank's review and approval of lending projects. PADs include a project rationale, key components of
the loan, potential areas of risk, and an assessment of the economic merits of the project.
4
We found that project documents often identify multiple investment options but present
detailed evaluations only for the already-preferred investment. The alternative options are
rarely analyzed quantitatively or compared meaningfully. It is possible that analysts
conducted detailed analyses prior to the project approval stage to whittle the options. Our
conversations with project leaders suggest that the preferred investment is often chosen
through an informal process and dialogue, not through a quantified analysis of costs and
benefits. Sometimes, the results of a Technical Assistance phase inform investment lending,
but this does not substitute for the lack of a clear quantification of risks. This confirms the
Independent Evaluation Group’s observation that economic analyses are in practice used to
justify rather than inform investment decisions, because they are done once the project is
already well advanced (IEG, 2010).
Projects often describe many deep uncertainties in the project narrative and rationale. For
instance, of the ten projects in our sample, eight note that population growth and nine note
that future socio-economic developments may shape the success of the proposed project.
Seven cite the uncertainty of energy supply and demand, four cite the uncertainty of urban
development, and three cite the worsening of environmental conditions as relevant to future
project outcomes.
However, in most projects (six out of ten) these external drivers are not included in the
economic analysis. Only two projects include considerations of future water availability in
their cost-benefit analysis. Another two projects include socio-economic developments in
their estimation of energy and transport investments, respectively. However, even when
economic analyses include these external conditions, they are not treated as deep
uncertainties but rather as well understood parameters, using single best estimate
projections.
There is also a disconnect between the risk section and the economic analysis in the PAD:
most of the potential risks mentioned in the risk section are not examined in the economic
analysis for the evaluation of the available options. Analysts address risks qualitatively, often
by proposing monitoring and capacity building. In the sample projects analyzed, none of the
risks mentioned in the risk analysis are carried through to the economic analysis.
To address uncertainty, projects assess the sensitivity of only the preferred option and with
respect to only a few of the many relevant uncertainties. Nine out of ten projects assess the
sensitivity of the chosen option to investment costs, yet only one tests its sensitivity to the
discount rate. Moreover, projects varied the uncertainties by a small amount around their
best estimate projection – often only 20%. This provides a very limited understanding of the
vulnerabilities of the project.
Moreover, many deep uncertainties that could clearly affect the success of a project are
never mentioned. For instance, the timely completion of a project may significantly affect the
project outcomes. All of the projects we reviewed mention this as a key risk but only two
included project delays in the economic analysis even though delays would affect the
projects’ rates of return. Similarly, the success of many investment projects depends on
effective future operations and maintenance. But, particularly in developing countries,
operations and maintenance of large infrastructure often falls short of needs (Foster et al.,
2010; Ostrom et al., 1993). Yet many projects assume operations and maintenance will
continue as planned – or that inefficiencies will be manageable - and the economic analysis
5
does not consider how the investment would perform with degraded operations and
maintenance.
Despite these shortcomings, two of ten projects do recognize that deep uncertainties affect
their choices. Instead of Predict-Then-Act approaches, they seek to manage uncertainty with
a handful of diverse scenarios that challenge prediction-based thinking. Scenarios can
usefully encourage decision makers to assess their investment options under unexpected
conditions.
However, scenario planning also struggles to manage deep uncertainties. Analysts usually
only consider a small number of scenarios. Analysts also typically handcraft the scenarios,
so they are subject to the same biases as analysts’ predictions about an expected future.
Making decisions based on diverse scenarios may also lead us again to a problem of
consensus around predictions – to which scenario should we tailor our investment decision?
Collectively, these shortcomings suggest that there is room for improvement to meet the
Independent Evaluation Group’s recommendations of rigorous economic analysis.
3 Robust Decision Making
Robust Decision Making is a decision support methodology designed to help manage deep
uncertainty by helping develop plans that are robust and flexible (Lempert et al., 2003). It
has been applied to water resource management, (Groves et al., 2008) flood risk
management (Fischbach, 2010), terrorism risk insurance (Willis et al., 2005), and energy
investments (Popper et al., 2009), primarily in the U.S. Lempert et al. (2013) provide a
concise summary of RDM and several case studies.
RDM is one of several methods like InfoGap (Ben-Haim, 2006) and Climate-Informed
Decision Analysis (Brown, 2010), that seek to better understand how decision makers’
options perform under a wide range of conditions, rather than under a single or handful of
predicted conditions. The World Bank recommends RDM and similar approaches for
managing uncertainty and making better-informed decisions (The World Bank, 2014). In one
of the first applications in developing countries, a recent World Bank study demonstrates
how RDM could help make robust flood risk management plans in Ho Chi Minh City
(Lempert et al., 2013).
RDM involves four basic steps that are embedded in a process of stakeholder engagement:
1. Decision makers structure the decision problem to identify their potential options,
the metrics and performance thresholds they will use to evaluate whether their
options meet or fail to meet their goals, and the uncertainties that could affect the
performance of the policy option according to the metrics. Analysts use models to
relate these factors.
2. Analysts statistically generate hundreds of futures and evaluate the performance
of their options in each of those futures. This generates a large table of inputs
(the uncertain future conditions) and outputs (the metrics) for each option.
3. Analysts identify which input conditions best explain when each option meets or
fails to meet decision makers’ performance thresholds. These conditions describe
scenarios to which each option is vulnerable.
6
4. Analysts and decision makers together compare the scenarios with available
evidence to determine if they are sufficiently plausible to hedge against. They
compare trade-offs between robustness, feasibility, cost, and other factors and
select those options that best balance their needs.
Analysts and decision makers iterate upon earlier steps to examine more options or modify
features of options, explore a wider range of uncertainties, and consider additional metrics.
As Lempert et al. (2013) describe, this approach can be used to time investments and
develop flexible plans – ones designed to evolve as new information becomes available.
RDM can be used to design individual projects, portfolios of projects, or to compare different
exclusive alternatives, as we do in this study.
4 Making Robust Energy Investments in Turkey
In 2006, the Electricity Generation Rehabilitation and Restructuring Project sought to
improve Turkey’s energy security in the near and mid-term. Turkey’s electricity demand had
grown rapidly between 2002 and 2005 – 6% on average -- and it was expected to continue
to grow, potentially outstripping its supply (The World Bank, 2006). The country had issued
licenses for about 6,000MW of new capacity, yet in 2006 little new construction had begun.
The Electricity Generation Rehabilitation and Restructuring Project focused on solving the
near term solutions as well as more systemic solutions for the medium term. In the near term,
it sought to increase electricity production through supply investments; for the medium term,
it sought to provide support for restructuring the sector.
The project examined eight different supply investment options for the Afsin Elbistan
electricity production area: seven types of power plants and the import of energy from
Bulgaria. They evaluated these projects based on two economic metrics. First, it should
pass a cost-minimization criterion, producing electricity at lower cost than any of the other
options. Second, the investment should pass a cost-benefit test; in this case, it should have
at least a 12% internal rate of return (IRR), a benchmark commonly used at the World Bank
to demonstrate economic value of investments.
At the time, decision makers originally preferred rehabilitating the existing local lignite-fired
plant at Afsin Elbistan because they believed it would produce electricity the soonest. The
project’s economic analysis further suggested that it would meet both their cost-benefit and
cost-minimization targets. The project was approved in 2006. However, it was never
implemented due to legal dispute with a private company, which had signed a concession
agreement with the government in 1999 for the operation of the plant and the eventual
construction of a new energy production plant.4
We use this project to demonstrate RDM for several reasons. The energy sector is fraught
with deep uncertainties that challenge decision making. It is also one of the most important
sectors for long-term economic growth, and simultaneously it can shape the development
4
The litigation was on-going at the time of project design, but both parties seemed willing to find a
solution. Hence, the risk of delays (but not of failures) was considered as moderate in the Project
Appraisal Document (The World Bank, 2006, p.11), though such delays were not included in the
economic analysis.
7
path of a nation. Additionally, this particular project has detailed data and a model that
enabled us to repeat the earlier economic analysis using RDM.
4.1 Summary of the Original Economic Analysis
The project’s original economic model evaluated seven power plants’ options for the Afsin
Elbistan area for increasing Turkey’s electricity supply, but five were addressed in detail 5:
1. Rehabilitating the existing local lignite-fired power plant;
2. Building a new local lignite-fired power plant;
3. Building a new imported coal-fired power plant;
4. Building a new gas-fired power plant; and
5. Building a new lignite fluidized bed power plant, which would be fired by a higher
quality coal than local lignite.
We replicated the analysis for these five investment options. Table 1 summarizes the
uncertainties examined in the project’s original analysis, and in our application of RDM to the
project. These relate to future energy conditions, future power plant investment
characteristics, and economic parameters.
The original analysis evaluated each option under a best estimate of several deep
uncertainties, listed in column A of Table 1. Analysts’ projected or assumed values are listed
in column B and were based on the data available at the time. For instance, the assumed
price of local lignite (6.05 US$/ton) is the levelized economic cost over a 20 year period from
the expanded and upgraded local lignite supply mine (The World Bank, 2006).
The original analysis prioritized options according to their cost-minimization, specifically the
discounted cost per unit of production. It showed that rehabilitating the existing plant would
be the most cost effective at 3.88 cents US$/kWh, followed by constructing a new gas-fired
combined cycle plant at 4.41 cents US$/kWh.
The project also sought to ensure that the selected investment would pass a cost-benefit
test, specifically having an IRR of at least 12%. It showed that rehabilitating the existing
plant would have an IRR of 25% under the projection in column B, comfortably passing the
12% target. To address uncertainty, the project conducted a sensitivity analysis of the IRR to
some of the projected values (column C). These values reflect 10%, 20%, or 50% deviations
from the projected value. The IRR was at least 14% across all the sensitivity tests. 6 This
analysis suggested that the rehabilitation project passes decision makers’ economic tests of
being the most cost effective option while also having high returns in the analysts’ best
5
A wind and a nuclear plant were also considered. The wind plant was eliminated as a candidate
because its cost effectiveness, which was derived from the literature, was not competitive with other
options. The nuclear plant was eliminated because it would take much longer to construct.
6
These results are reported directly from the project appraisal document. We reconstructed the
economic model using the same formulae and data as the original model, but with slightly different
time series of the price of inputs, which were difficult to replicate from the original analysis. Our model
results showed the same relative performance of options. However, running the original sensitivity
analysis for rehabilitating the plant in our model revealed that the IRR fell below12% when electricity
prices were 4 cents US$/kWh, or when the cost of lignite is 9 US$/ton. This does not suggest that the
original sensitivity analysis is incorrect, but rather that the results can be highly sensitive to slight
variations in model parameters. A small sensitivity analysis may not reveal the full behaviour of the
interventions under analysis.
8
estimate of future conditions. The project’s economic analysis concludes that “[…] the
rehabilitated plant will be the least cost option for Turkey under any scenario […]” (The
World Bank, 2006:58).
Table 1. Uncertainties evaluated in the original analysis and in this study’s RDM analysis.
Original Analysis RDM Analysis
A. Deep Uncertainties B. Projected C. Sensitivity D. LHS Range
b
Value Test Values
min max
a
Wholesale price of electricity (US$/kWh) 0.05 0.04, 0.045 0.04 0.10
a
Discount Rate 0.1 — 0.01 0.20
Estimated Life of the Plant (years)
Rehabilitation; New local lignite plant;
20 (10, 16) 5 25
New gas-fired plant;
New imported coal plant; New lignite
40 — 5 45
fluidized bed plant
Capacity Utilization
Rehabilitation; New gas-fired plant; New
0.76 (0.6, 0.68) 0.5 0.9
lignite fluidized bed plants
New local lignite plant; New imported
0.90 — 0.6 0.95
coal plant
7
Capital Costs (US$, in millions)
Rehabilitation 683 (819, 1024.5) 600 1030
New local lignite plant 1998 — 1800 3000
New imported coal plant 586.7 — 500 880
New gas-fired plant 420 — 350 630
New lignite fluidized bed plant 451.84 — 400 670
Length of Construction Time (years)
Rehabilitation 4 — 2 6
New local lignite plant 6 — 3 9
New imported coal plant; New gas-fired
3 — 1.5 4.5
plant; New lignite fluidized bed plant
Cost of energy inputs
Local lignite (US$/ton, for the
6.05 (7.06, 9.08) 3 12
rehabilitated and the new plant
Imported coal (US$/ton) 60 — 30 120
Gas (US$/tcm, for the new gas-fired
220 — 110 440
plant)
8
Improved coal for the new lignite
21 — 10 42
fluidized bed plant (US$/ton)
a
Only the wholesale price of electricity and the discount rate have the same ranges across all options’
cost effectiveness models, both in the original and in the RDM analyses.
b
The original analysis conducted sensitivity tests only for the IRR of rehabilitating the existing plant.
7
Project capital costs included two key technological improvements. Given the poor quality of local
lignite, both the rehabilitated and the new local lignite-fired power plants would have flue-gas
desulfurization (FGD) to remove sulfur dioxide from exhaust flue gases. During project preparation,
dust reduction technology was also made compulsory in all options with low quality coal.
8
The three types of coal – local lignite, coal for the fluidized bed plant, and the imported coal -- have
a different energy values. Local lignite has the least and imported coal has the most potential energy.
9
4.2 Demonstration of Robust Decision Making
Robust Decision Making can help us more fully understand the strengths and weaknesses of
our options and make a sound decision, without relying on accurate predictions of the
unpredictable future. It helps us answer several useful questions:
1. How do our options perform across a wide range of potential future conditions?
2. Under what specific conditions does the rehabilitating the existing plant fail to
meet our goals?
3. Are those conditions sufficiently likely that we should choose a different energy
investment option?
In practice, these questions are addressed in a close collaboration between decision makers,
stakeholders, and analysts. A series of structured workshops help participants frame the
analysis, deliberate over interim findings and direct subsequent analyses, and reach
decisions. Because we are re-creating an earlier project analysis, we focus in this section on
the analytical steps and draw on the outcomes of stakeholder engagement in the original
project.
Structuring the Problem
To conduct the RDM analysis, we first structure our analysis to more fully identify the
uncertainties we face. In this study, we used the same uncertainties as in the original
analysis, but in nearly all cases, we have expanded the range of possible values as shown
in column D of Table 1. Figure 1 shows this visually for the option of rehabilitating the
existing plant. The values’ ranges were drawn from the literature and historical data, and
from consultations with energy experts. For example, literature on investment planning
biases suggests that on average, capital costs are underestimated by 28% (Siemiatycki,
2010). World Bank energy experts confirmed that for a comprehensive thermal rehabilitation
project, cost overruns of 50% are possible. Hence, we considered a range of capital costs
from a lower bound of just below each option’s stated cost, to an upper bound of 150% of
the stated cost.
We also added two new uncertainties: the length of construction time and discount rate. The
length of construction time is an uncertain and important parameter. Most projects
experience delays in implementation (Ahsan and Gunawan, 2010). Indeed, the original
analysis identified possible delays in rehabilitation and construction as plausible risks.
Moreover, decision makers were concerned with how quickly each option could be
implemented and would begin producing electricity.
The discount rate is also deeply uncertain. It is a political choice and often highly-
contested (Arrow et al., 2013). It shapes how we allocate resources between the present
and the future (Gollier, 2011). A higher discount rate signifies an urgency to satisfy present
needs, whereas a lower discount rate expresses concerns for the long-term impacts of an
investment. Although the World Bank typically uses discount rates of 10% to 12%, no single
discount rate is appropriate for all projects and it may be difficult for stakeholders to come to
consensus (Hoekstra, 1985; Oxera, 2011). After consultation with World Bank experts, we
use a range from 1 to 20% to explore both the longer-term considerations of contributing to
the country's growth and the short-term objectives of avoiding an energy crisis.
10
Figure 1 Uncertainty space assessed by the projection, sensitivity analysis, and RDM analysis
of rehabilitating the existing plant.
The original sensitivity tests were only carried out for the cost-benefit metric of rehabilitating
the existing plant and variables were made to vary one at a time. In contrast, we conduct the
RDM analysis for all options and for both the cost-benefit and cost-minimization metrics.
Note that we are not assigning any likelihood to values in this range. We use the ranges to
answer the question, “What could the future bring and how would it affect our investment?”
rather than “What will the future bring?”
Generating Futures
We then statistically generate 500 futures, each a combination of one value for each
uncertainty 9. Again, these futures are not predictions, and we do not assign any likelihood to
their occurrence. We use them to better understand the behavior of our investment options.
We evaluate the cost per kWh of energy (the cost-minimization criterion) and the IRR (the
cost-benefit criterion) of each of the five options in each of these 500 futures. 10 The result is
a table of 2,500 model runs, four rows of which are shown in Table 2. This table of results
helps us answer our key questions.
9
In this study, we use Latin Hypercube Sampling, which is similar to but more efficient than Monte
Carlo sampling. It examines the behaviour of our options over the full range of uncertainties with the
fewest number of samples.
10
To easily repeat the analysis, we re-implemented the economic analysis equations from Excel into
the Analytica risk modeling environment. Analytica is a visual modeling platform for quantitative risk
and uncertainty analysis. It allows analysts to create influence diagrams that define how factors in
analysis relate to each other and to quickly add or modify elements of the model during the course of
the analysis and in response to input from stakeholders. Analytica is well suited for managing
uncertainty because, unlike spreadsheets, it can be easily configured to run over many futures and
save the results to a table. See www.lumina.com.
11
Table 2. Sample from the database with the 500 futures
Inputs Outputs
Wholesale Price of
Construction time
Discounted Cost
Uncertainties*
Capital Cost
Other four
Electricity
Length of
Future ID
per kWh
Option
IRR
1 Rehabilitation 5.15 824.89 2.85 […] 6.39 0.05
2 Rehabilitation 8.79 945.29 4.14 […] 7.11 0.35
1 Gas-fired plant 5.15 526.68 4.01 […] 4.21 0.27
2 Gas-fired plant 8.79 514.92 4.38 […] 8.91 0.15
* Other uncertainties are the discount rates, cost of inputs, lifetime of the plants, and capacity
utilization rates.
How do our options perform across a wide range of potential future
conditions?
Figure 2 shows the performance of the investment options in 500 plausible futures.
Rehabilitating the existing local-lignite plant meets the cost-minimization and cost-benefit
targets in most futures. However, there are many in which constructing a new gas-fired plant
would be more cost effective. There are also some futures in which the rehabilitation option
has an IRR of less than12%. The other three options meet performance goals in far fewer
futures. This indicates that rehabilitation meets our goals under a wider range of
assumptions about the future than do other options.
Up to this point, our analysis resembles the steps in traditional Monte Carlo analyses used
for project evaluation: 11 both approaches run many simulations over randomly generated
cases. However, the approaches diverge significantly in how they use such simulations. A
Monte Carlo analysis uses the sampled cases to represent the likelihood of future conditions
(e.g. probabilities of future energy prices, construction time, etc.). It further uses the results
of the simulations to make inferences about the likely performance of the project, e.g. that
rehabilitation is more likely to meet decision makers’ goals than other interventions because
it outperforms other interventions in most cases. This approach works well and the
inferences are credible when we have reliable probability distributions.
However, in many investment decisions such as this one, we do not have defensible
probability distributions of our deep uncertainties and therefore we cannot infer the relative
success or failure of our options from these results. For example, although rehabilitation
produces energy at lowest cost in 269 of 500 futures, this does not mean that the probability
of doing so is 54% (269/500). Similarly, although rehabilitation falls short of a 12% IRR in 99
11
Monte Carlo analysis describes a useful approach to numerically generating a probability
distribution of an outcome by repeatedly and randomly sampling input parameters from probability
distributions of their values (Belli, 2001). For project evaluation, analysts first assign probability
distributions to inputs such as future energy prices, length of project implementation, climate change
impacts. The resulting distribution of outcomes (e.g., IRR) is interpreted as a statement of the
project’s likelihood of success or failure.
12
of 500 futures, this does not mean that the probability of failing to meet this threshold is 20%
(99/500).
Instead, in an RDM analysis, we sample uniformly across the range of plausible values of
our deep uncertainties to ensure that we represent all viewpoints about the future in our
analysis. This does not mean that we believe that each case is equally likely. Indeed, we are
making no statements about likelihood at all. Rather, we use the cases to stress test the
performance of a project over the widest range of possible conditions. Then, in the next step,
we mine the database of simulation results to identify the specific set of underlying
conditions that lead each option to fail to meet our goals. Finally, we assess the relative
plausibility of these threatening conditions to determine which option is more robust and to
present trade-offs to decision makers.
Figure 2. The cost-effectiveness and cost-benefit performances of the five investment options
in 500 futures.
Under what specific conditions does the leading option fail to meet our
goals?
We next used statistical “scenario discovery”, running data-mining algorithms on the table
of results. 12 This step identifies the combinations of uncertain future conditions that most
reliably distinguish those futures in which rehabilitation of the existing plant does not satisfy
the cost-minimization or cost-benefit targets, from those futures where it does. This step,
typically performed by analysts and modelers, identifies the key drivers of the decision and
focuses attention on those future conditions that would matter given the investment options
available.
12
We used the Patient Rule Induction Method, which is available as a free software package in the R
programming environment. http://cran.r-project.org/web/packages/sdtoolkit/sdtoolkit.pdf
13
Scenario discovery reveals that a gas-fired plant is more cost effective than rehabilitating
the existing plant if:
1. The cost of local lignite (US$/ton) is more than 4.5% of the cost of gas
(US$/tcm). 13
Scenario discovery also reveals that rehabilitating the existing plant fails our cost-benefit
test, i.e. has an IRR below 12%, if two conditions hold simultaneously:
1. The wholesale price of electricity is below 0.059 US$/kWh
2. Local lignite costs more than 6.3 US$/ton14
The discount rate, estimated life of the plant, capacity utilization, length of construction time,
and capital costs – though highly uncertain and potentially a source of disagreement among
stakeholders -- are less important in determining whether or not investing in rehabilitating
the plant is economically sound. We could invest significant time discussing what these
values should be, when indeed they are not key drivers of the economic performance of our
investment.
Are those conditions sufficiently likely that we should choose a different
energy investment option?
So far we have sought to better understand the merits and drawbacks of our options under
a wide range of conditions, and to identify the conditions that may lead them to fall short of
our goals. We have said little about what the future may actually hold. We now turn to the
evidence to assess whether those conditions are sufficiently likely that we should choose a
different energy investment option. We begin by examining the vulnerabilities to cost-
minimization and then to cost-benefit goals.
Recall that rehabilitating the existing power plant is less cost effective than building a new
gas-fired plant if the cost of local lignite (US$/ton) is more than 4.5% of the cost of gas
(US$/tcm). The original cost-benefit analysis assumed that with an upgrade, the mine would
sell local lignite at an average of 6.05 US$/ton, 2.8% of the 2006 gas price of 220 US$/tcm
and well below the 4.5% threshold. 15
However current and historical trends do not guarantee or bound future outcomes, and we
should not use them to predict what will be. Instead, we should use them to answer a more
useful question, “Is there evidence that the conditions that threaten the success of our
13
This condition is a statistically strong predictor of when building a new gas-fired plant is more cost-
effective than rehabilitating the existing plant. Of the 97 futures with this condition, building a new
gas-fired plant is more cost effective in 92 of them (95%). However, it is not a complete predictor:
this condition exists in 92 of the 153 futures (60%) in which the new gas-fired plant is more cost
effective. Further scenario discovery analysis, beyond the scope of this demonstration, would reveal
additional sets of conditions that explain the remaining 40%, which policy makers could weigh
against additional evidence. Nevertheless, this single condition offers useful information for a policy
dialogue on the potential vulnerabilities of the rehabilitation option.
14
This condition is also a statistically strong predictor of when rehabilitating the existing plant fails
the cost-benefit test. Of the 92 futures with these conditions, rehabilitation fails the cost-benefit test in
73 (79%). Of the 99 cases in which rehabilitation fails the cost-benefit test, this condition occurs in 73
(74%).
15
In the time series utilized by the project, the ratio maintains an average of 3.1% and never
exceeds the 3.5% thresholds (The World Bank, 2006).
14
investment could occur?”
The answer is yes. The cost of local lignite depends substantially on the efficiency and
output of the mine. Let us turn briefly to the main drivers of local lignite costs. The cost of
local lignite is at least partially under decision makers’ control. It depends on the efficiency
of the mine that supplies lignite to the plant, i.e. how many millions of tons of lignite are
extracted annually. Most of the mine’s costs, including labor, electricity, and materials, are
fixed and when production is down, prices are high. At low production in 2004, just before
the project’s preparation phase, the mine produced only 6.7M tons, about 30% of the
mine’s annual capacity of 18.6M tons, which were sold at 11.46 US$/ton (The World Bank,
2006). As part of this project, however, this mine would have been expanded and upgraded
to ensure a cheaper and more stable lignite supply. Hence, policy makers have options to
reduce the cost of lignite, for instance by improving the mine’s maintenance and operation.
From 2002 until 2008, gas prices increased sharply in Turkey, reaching over 300 US$/tcm
(Ozen, 2012) but have since declined and stabilized at around 220 US$/tmc (DG Energy,
2012; Lomsadze, 2013). Hence, the highest ratio based on recent historical trends, when
lignite cost is high (11.46 US$/ton) and gas price is low (220 US$/tmc), would be 5.2%. This
exceeds the threshold of 4.5%. Indeed, if current trends continue to hold and the price of
gas remains low, the cost of local lignite would have to be below 9.9 US$/ton (i.e. produce
at least at about half of its capacity) to be under the 4.5% threshold (9.90/220).
It thus appears that rehabilitating the existing plant may not be the most cost-effective
option under all plausible future conditions, though policy makers have options to influence
those conditions.
We can more fully understand the relative merits of the two investments by also comparing
their cost-benefit performance. Recall that rehabilitating the existing plant fails our cost-
benefit test in 99 of 500 futures, i.e. has an IRR below 12%, when two conditions hold:
1. The wholesale price of electricity is below 0.059 US$/kWh
2. Local lignite costs more than 6.3 US$/ton16
Building a new gas-fired plant fails our cost-benefit test in 128 of our 500 scenarios, which
occurs when:
1. The wholesale price of electricity is below 0.073 US$/kWh
2. Gas costs more than 220 US$/tcm 17
Rehabilitating the existing plant would be more robust to decreases in the price of electricity
than would the gas-fired plant. In 2009, the Government of Turkey guaranteed a minimum
price of 0.068 US$/kWh for the following 10 years (U.S. Commercial Service Turkey, 2009).
This price is above the vulnerable threshold for rehabilitating the existing plant, but below
16
This condition is also a statistically strong predictor of when rehabilitating the existing plant fails
the cost-benefit test. Of the 92 futures with these conditions, rehabilitation fails the cost-benefit test in
73 (79%). Of the 99 cases in which rehabilitation fails the cost-benefit test, this condition occurs in 73
(74%).
17
This condition is also a statistically strong predictor of when building a new gas-fired power plant
fails the cost-benefit test. Of the 125 futures with these conditions, rehabilitation fails the cost-benefit
test in 100 (80%). Of the 128 cases in which rehabilitation fails the cost-benefit test, this condition
occurs in 100 (78%).
15
the threshold for the gas-fired plant. Nevertheless, in 2009, Turkish electricity’s wholesale
prices were approximately 0.10-0.11 US$/kWh and were not likely to decrease for years to
come until excess capacity was in the marketplace.
As noted above, policy makers have options to reduce the cost of lignite. In particular,
operating the mine at about 13 million tons per year -- roughly two-thirds capacity -- would
keep the lignite cost below 6 US$/ton (The World Bank, 2006). Simultaneously, the cost of
gas has remained above 220 US$/tcm since 2006. In combination, the conditions that
would lead the gas-fired plant to fail the cost effectiveness test seem more plausible than
the conditions that would lead rehabilitation to fail the cost-benefit test.
In sum, this analysis leads to several important insights about the merits of our options:
1. Rehabilitating the existing lignite plant fails to meet decision makers’ cost-
minimization goals under plausible future conditions. If gas costs remain low
while local lignite costs increase, a new gas-fired plant would be more cost
effective.
2. Yet, the cost of local lignite is at least partially under decision makers’ control,
suggesting that decision makers can take action to avoid those threatening
conditions.
3. Moreover, building a new gas-fired plant fails to pass the cost-benefit test under
conditions that are less constraining and more plausible than the conditions in
which the rehabilitation fails a cost-benefit test.
These observations could reasonably lead decision makers to conclude that rehabilitating
the existing plant is an economically sound choice: it performs well under a wide range of
potential future conditions and is less vulnerable than other options. However, other factors
may also influence their decision. The relative importance of cost-benefit versus cost-
effectiveness metrics, decision makers’ risk aversion, and other concerns may shape their
choice. Nevertheless, a more complete and more transparent understanding of the merits
of decision makers’ options helps inform these deliberations and keeps the decision in the
hands of decision makers.
5 Key Findings and Recommendations
Faced with pervasive deep uncertainties about the future, decision makers struggle to
choose near-term decisions that make long-term sense. In this paper, we sought to answer
three key questions:
1. How are deep uncertainties currently managed in World Bank projects?
2. Can new approaches help projects better manage those uncertainties?
3. Are they practical and applicable, and what challenges do they pose?
To help answer these questions, we reviewed prior project analyses and then applied RDM
to the economic analysis of a long-term energy investment in Turkey. Our goal is not to
reevaluate the earlier decision, which today would be very different and for example,
include climate change concerns. Rather, we show the type of questions decision makers
and analysts should ask about the robustness of their investments, and how they can
16
answer them. Through this demonstration, we also hope to motivate and equip analysts to
better manage uncertainty in investment lending decisions.
Current World Bank Project Analyses Struggle to Manage Deep Uncertainty
Our review of prior project analyses suggests that analysts typically use traditional Predict-
then-Act methods to assess project performance, and do not explore the full range of deep
uncertainty. In particular:
1. Project documents contain efforts to manage uncertainties.
2. However, even though project narratives may identify several deep
uncertainties, the economic analyses address only a few.
3. There is a disconnect between risk and economic analysis. For instance,
analyses typically assume that project costs and implementation will occur as
planned, even though implementation issues are often cited as potential risks to
the project.
4. Projects often carry out sensitivity analyses only for the already-preferred
investment option or for the option shown to perform best under a single
prediction of future conditions.
5. Most project managers we contacted during this study are aware of the
potential for improving economic analysis.
In sum, project analyses can be improved to help make projects more robust.
Methods Like RDM Can Help Make Better-Informed Investment Decisions
The findings from the Turkey project suggest that methods like RDM can improve our
understanding of the vulnerabilities and strengths of our investment options, despite deep
uncertainty and a potentially surprising future. In particular, this analysis demonstrates how
RDM can help analysts and decision makers
• Identify and make explicit the many deep uncertainties that may affect the
performance of investment options;
• Analyze the performance of investment options across a full range of plausible
futures, without needing to assign controversial likelihoods to those futures;
• Identify and focus attention on the specific combinations of conditions that
determine whether an investment meets or fails to meet decision makers’ goals;
and
• Use historical and scientific evidence credibly, to assess whether threatening
conditions are plausible, rather than to make predictions about what will be.
Such knowledge informs rather than replaces decision makers’ deliberations. It helps them
systematically, rigorously, and transparently compare their options and select one that is
robust – meeting their needs in the widest range of possible futures. Decision makers can
have confidence in a robust decision, even if they cannot have confidence about the future.
17
These Methods Can Be Readily Applied, Though They Pose Some Challenges
In this study, we used the same economic models and data that analysts used in the
original analysis; we just used them differently. Rather than using the data and models to
assess performance in a single best estimate of the future, we used them to stress test our
options in hundreds of possible futures. This approach was more forgiving of data gaps:
where a quantity was unknown, we could use a wide range of plausible values, rather than
tenuously choosing a single value. This also enabled us to include uncertainties that may
not have been feasible in the original analysis.
Nevertheless, methods like RDM may have a steep learning curve – from understanding
how to structure a robustness analysis, to learning software that aids in scenario discovery,
to interpreting the results of scenario discovery, to communicating the findings to
stakeholders. However, prior applications of RDM with other organizations and agencies
suggest that these challenges can be readily overcome with time and training.
Managing Uncertainty Requires Rethinking “Good” Decision Making
Analysts and decision makers routinely face pressures to demonstrate that a decision is
risk-free. Political and cultural expediency press them to ignore rather than acknowledge
uncertainty and present their decision as advantageous and certain. Such thinking keeps us
in the dark about the real threats to our decision, and may lead us to brittle decisions that
fail when the future surprises us.
To manage uncertainty, we may need to revisit our beliefs about what makes a decision
“good.” Instead of ignoring uncertainty, we should seek more fully to understand the threats
it may pose to our choices. This will enable us to make decisions that are robust to an
unpredictable future.
Such a change requires a cultural shift as much as it requires an analytical shift. Yet
methodological innovations like RDM can help. By motivating and equipping analysts to
manage uncertainty, they can shape how we think, discuss, and ultimately make decisions.
18
References
Ahsan, K., Gunawan, I., 2010. Analysis of cost and schedule performance of international
development projects. Int. J. Proj. Manag. 28, 68–78.
Arrow, K., Fisher, A., 1974. Environmental preservation, uncertainty, and irreversibility. The
Quarterly Journal of Economics 88, 312–319.
Belli, P., 2001. Economic Analysis of Investment Operations: Analytical Tools and Practical
Applications. World Bank Publications.
Ben-Haim, Y., 2006. Info-Gap Decision Theory: Decisions Under Severe Uncertainty.
Academic Press.
Brown, C., 2010. Decision-scaling for Robust Planning and Policy under Climate
Uncertainty., World Resources Report. World Resources Institute, Washington, DC.
Coombes, Y., Devine, J., 2010. Introducing FOAM: A Framework to Analyze Handwashing
Behaviors to Design Effective Handwashing Programs (Working Paper), Global
Scaling Up Handwashing Project. Water and Sanitation Program.
Dessai, S., Wilby, R., 2010. How Can Developing Country Decision Makers Incorporate
Uncertainty about Climate Risks into Existing Planning and Policymaking
Processes?, Uncertainty Series. World Resources Report.
DG Energy, 2012. Quarterly Report on European Electricity Markets ( No. Volume 5, Issue
1), Market Observatory for Energy.
Fischbach, J.R., 2010. Managing New Orleans Flood Risk in an Uncertain Future Using
Non-Structural Risk Mitigation [WWW Document]. URL
http://www.rand.org/pubs/rgs_dissertations/RGSD262.html (accessed 11.1.13).
Foster, V., Briceño-Garmendia, C., World Bank, Africa Infrastructure Country Diagnostic,
2010. Africa’s infrastructure: a time for transformation. Agence Française de
Développement : World Bank, Washington, DC.
Groves, D.G., Knopman, D., Lempert, R.J., Berry, S.H., Wainfan, L., 2008. Presenting
Uncertainty About Climate Change to Water-Resource Managers [WWW Document].
URL http://www.rand.org/pubs/technical_reports/TR505.html (accessed 11.1.13).
Hallegatte, S., Shah, A., Lempert, R.J., Brown, C., Gill, S., 2012. Investment Decision
Making Under Deep Uncertainty: Application to Climate Change. Washington, D.C.,
World Bank.
Henry, C., 1974. Investment decisions under uncertainty: the“ Irreversibility Effect”. The
American Economic Review 64, 1006–1012.
Hoekstra, D.A., 1985. Choosing the discount rate for analysing agroforestry
systems/technologies from a private economic viewpoint. For. Ecol. Manag. 10,
177–183.
IEG, I.E.G., 2010. Cost-Benefit Analysis in World Bank Projects. The World Bank.
Independent Evaluation Group (IEG), 2012. Adapting to Climate Change: Assessing World
Bank Group Experience. Phase III of the World Bank Group and Climate Change.
The World Bank.
Kolb, D.A., 1984. Experiential learning: experience as the source of learning and
development. Prentice Hall, Englewood Cliffs, NJ.
Lempert, R., Kalra, N., 2011. Managing Climate Risks in Developing Countries with Robust
Decision Making, Uncertainty Series. World Resources Report.
Lempert, R., Kalra, N., Peyraud, S., Mao, Z., Sinh Bach, T., Cira, D., Lotsch, A., 2013.
Ensuring Robust Flood Risk Management in Ho Chi Minh City. World Bank, Policy
Research Working Paper.
Lempert, R.J., Popper, S.W., Bankes, S.C., 2003. Shaping the Next One Hundred Years :
New Methods for Quantitative, Long-term Policy Analysis. RAND Corporation, Santa
Monica, CA.
Lempert, R.J., Popper, S.W., Groves, D.G., Kalra, N., Fischbach, J.R., Bankes, S.C.,
Bryant, B.P., Collins, M.T., Keller, K., Hackbarth, A., Dixon, L., LaTourrette, T.,
19
Reville, R.T., Hall, J.W., Mijere, C., McInerney, D.J., 2013. Making Good Decisions
Without Predictions ( No. RB-9701), Research Brief. RAND Corporation.
Lomsadze, G., 2013. Turkish Coal Imports.
Ostrom, E., Schroeder, L.D., Wynne, S.G., 1993. Institutional incentives and sustainable
development: infrastructure policies in perspective. Westview Press.
Oxera, 2011. Discount rates for low-carbon and renewable generation technologies.
Prepared for the Committee on Climate Change.
Ozen, E., 2012. Turkey’s natural gas market. Expectations and developments 2012.
Deloitte.
Popper, S.W., Berrebi, C., Griffin, J., Light, T., Daehner, E.M., Crane, K., 2009. Natural Gas
and Israel’s Energy Future [WWW Document]. URL
http://www.rand.org/pubs/monographs/MG927.html (accessed 11.1.13).
Ranger, N., 2013. Topic Guide. Adaptation: Decision making under uncertainty. Evidence
on Demand.
Rennkamp, B., 2012. Sustainable Development Planning in South Africa: A Case of over-
Strategizing?, in: Conference Paper of the 2012 Berlin Conference of the Human
Dimensions of Global Environmental Change on “Evidence for Sustainable
Development”. 5- 6 October 2012, Berlin.
Siemiatycki, M., 2010. Managing Optimism Biases in the Delivery of Large-Infrastructure
Projects: A Corporate Performance Benchmarking Approach. Eur. J. Transp.
Infrastruct. Res. 10, 30–41.
The World Bank, 2006. Project Appraisal Document for the Electricity Generation
Rehabilitation and Restructuring Project, Turkey.
The World Bank, 2013. Investment Project Financing Economic Analysis Guidance Note,
OPSPQ. The World Bank.
The World Bank, 2014. Risk and Opportunity. Managing RIsk for Development, World
Development Report. The World Bank, Washington, DC.
U.S. Commercial Service Turkey, 2009. Electricity Power Systems in Turkey. U.S.
Department of Commerce.
Willis, H.H., Morral, A.R., Kelly, T.K., Medby, J.J., 2005. Estimating Terrorism Risk [WWW
Document]. URL http://www.rand.org/pubs/monographs/MG388.html (accessed
11.1.13).
20