The Economist explains

Why doing a cost-benefit analysis is harder than it looks

WHEN it comes to assessing the viability of a large project, governments are increasingly likely to commission a cost-benefit analysis (CBA). The aim is to provide an impartial, evidence-based judgment of the costs and benefits of a particular policy or project, without regard to its political ramifications. The British government is particularly keen on CBAs: having pioneered their use in the 1960s it has recently relied on them to make the case for HS2, a controversial high-speed rail link between London, Birmingham and Manchester, and to justify a cull of badgers in the countryside. In theory, putting together a CBA is simple: you simply tot up the costs in one column and the benefits in another. But the reality is rather more complex. With large amounts of money at stake and projects that can last for decades, economists have to use a number of wonkish techniques, some of which are controversial, to come up with a decent CBA. How do they do it?

The simplest and most important concept is a dull economic term: “consumer surplus”. This is the difference between what you are willing to pay for something and what you end up paying. If you’re willing to spend £1 on an apple, but get it for 40 pence, your consumer surplus is 60 pence. Consumer surplus is important because big projects like railways are often not sustainable from a purely commercial perspective. In the case of HS2 one estimate puts the expected revenue from fares at £15 billion, but the overall costs at £25.5 billion. But that is potentially a narrow way at looking at the benefits of such a project. Governments can also look at what they are saving citizens. Consider the case of someone who is used to paying £100 to get from Manchester to London. If HS2 is built, they could pay £40. The commercial benefit of the project is a mere £40; but the CBA will take into account the £60 worth of consumer surplus as well (because that sum is, in effect, unlocked to be spent on other things). Big infrastructural projects often make economic sense only when consumer surplus is taken into account.

To make costs and benefits fully comparable, further economic trickery is needed. Adjusting for inflation is an obvious first step. Then you must convert the calculated costs and benefits at various times to values at a single point in time, so that they can be compared. Economists refer to this as “net present value”. Wonks also need to think about how money invested in a project might be better spent. The government could just choose to shove the money into a bank account and gather the interest, or invest instead in another project that offers higher returns. Economists call this idea the “opportunity cost of capital”. There are no golden rules for choosing the appropriate rate—sometimes called the “discount rate”, though it often corresponds to what people could get by buying government bonds. In an analysis of the badger cull the British government went for 3.5%. The higher the discount rate, the smaller the future benefits will seem. Some worry that those opposed to big investment projects deliberately exaggerate discount rates.

As CBA has become integral to large projects, the limitations to its methodology have come under greater scrutiny. CBAs struggle to put monetary values on things like environmental quality. Crafty economists try to get around this problem by calculating “willingness to pay”: working out how much money someone would spend to clean the air or purify water. In addition, no two CBAs are alike, so it is hard to compare different studies. And sometimes the assumptions can be heroic, to say the least. An early analysis of HS2 claimed that people did not do any work on trains, thereby increasing the benefit of shorter journey times. This idea was subsequently scrapped and the focus became more on the economic benefits to the north. But despite being controversial CBAs are popular. Governments will rarely approve a big project without first submitting their proposal for wonkish scrutiny.

CBA is harder than it looks? A simple Dutch advice: take it easy on the pricing, but worry about the Q!

Apparently the British government is just as keen on cost benefit calculations as the Dutch. And it makes sense because the alternative is to simply spend money on large projects without any idea of their welfare increasing potential. And yes, nobody ever said that putting together a CBA was easy. It is interesting, however, that everybody always says that pricing environmental qualities is hard. As if pricing travel time reductions, the most important benefit of transportation projects, is less hard.

Well here is some goods news, based on years of experience with CBA calculations: pricing is the easy part. Anyone can determine a consumer surplus, by doing a survey that measures peoples willingness to pay. And to make sure that CBA’s are all carried out in a comparable way, governments can publish authorised survey results. This way everybody uses the same prices. In the Netherlands we have authorised values of travel time, but also for all kinds of environmental impacts, such as air quality, noise, safety and yes, also for ecosystem services!

With authorized experience numbers pricing has become a ‘piece of cake’. Now, there is only one difficulty left: what is the correct quantity to multiply the price with? You many hours of travel time are saved? You many traffic accidents are prevented? What is the reduction in asthma patients due to improved air quality? The quantification of welfare impacts, that is the real issue of a CBA. There is no point in having a price, if you cannot get the quantity right.

For the quantification of impacts model forecasts can be used. E.g. traffic models are used to predict travel time reductions. These models can be black boxes and generate strange results. Here experience numbers can be an interesting alternative. For impacts, such as ecosystem services, quantification experience numbers are available. E.g. how much carbon can be fixated by an acre of forest, how much nitrogen can a reed vegetation remove from the water, how many recreational trips can a park accommodate etc. In the Netherlands we have a book of reference filled with both Q-numbers and prices (P’s) for the computation of impacts on the natural environment. Of course this book does not solve all CBA problems. There is plenty of room for improvement, but it’s a start. A start to focus on the Q instead of endlessly worring about the P.

I think the limitations on CBA are one problem and whether to use them another. It is an excellent innovation for governments to be mindful of costs and benefits even when some of each cannot be included in the equation.

CBA is clearly wrong because it's got the fundamentals, the ABC's, spelled back-to-front.

:-D

I am only half-joking: anyone who has done a CBA at a senior level will know that we can set a particular reasonable target result and then do a CBA which fully justifies that target result. So to say CBA is back-to-front is not as facetious as it seems.

this is why two-party democracy is important - to give two sides to every story - Australia currently has a new government which is breaking election promises to reveal itself as true conservative - cutting budgets and welfare payments to poor people, while allowing big business to continue to pollute and make huge profits while cutting jobs without scrutiny

without an active and participatory democracy, we can end up with your totalitarian dictatorship like North Korea where the smiles of the faces of women crowding around the Dear Leader are more likely - 'please don't kill me' and 'I'm starving - can I get a meal?'

Great start, but could you do a further article focusing on the values used in the HS2 analysis for the values you refer to - or provide a link to the report where the choices are stated and argued for?

As a long-time practitioner in budgeting and economic policy, both in the US in other countries, let me suggest a little reality for your consideration. Just as economists rarely look back at their earlier economic forecasts, systematic evaluation of past cost-benefit analyses are either rare or nonexistent. This should give us pause. The bridge over the Bosporus did not meet benefit cost standards according to development economists, yet not long after the first one was built, another one was required. In the United States the president's budget Bureau, now the Office of Management and Budget, reviewed plans for the Washington, DC metropolitan light rail Metro, and each year determined that it did not meet benefit costs standards. It was approved basically for political reasons. Yet today Washington is inconceivable without the system. Or consider benefit cost analysis of logging in the Western United States. In this case not only is the amount of benefit in dispute, but the sign (i.e. plus or minus) is not agreed upon. In Scandinavian example (if memory serves it was Finland) engineers and decision-makers were unable to think of any quantifiable objective way of measuring the benefit costs of a tunnel versus a bridge. Costs are harder not to estimate but benefits in realistic terms often become in ponderable. The inhabitants of our islands in the United States in both the Atlantic and Pacific opposed bridges in a number of cases because they want to ration the number of visitors by controlling the capacity of ferryboats. Suppose these bridges would, by objective economic measures, such as the future value of houses, yield the benefits, while at the same time the owners of the houses oppose the bridges? How do you deal with benefits of wind farms to people who worry about global warming versus detriments to people who think they're ugly?
D Mathiasen washington

Once again, Doug is right on the money. As a one-time exercise, CBAs are very subjective.

Why are we doing this (transport) project? In the embryonic/concept stages, it might be to reduce travel times but we can't quite get a handle on the environmental costs, nor the loss of economic production in the countryside, so they are 'externalities'.

At the feasibility stage, town planners have come on board and quantified the cost saving by decentralising housing and urban services. The agriculture/ horticulture experts have got a first cut of what the new farming structures might look like, with some ingenious workarounds that retain agricultural efficiencies. But environmental costs are still outside our grasp.

The next iteration will be project design, with a new look at the benefits and costs that are in/out. A great way to prevent scope-creep, and the project gets a lease of life if underpinned by multiple streams of unexaggerated benefits.