Flashback Saturday: Probing cost-benefit analyses

(Almost) every weekend we dig into the archives. This post was first published in October 2009.

A couple of months ago I stated that if there were two things I could change about transport, it would be creating one pot of money for transport – and also to change the way that transport projects are analysed. This post focuses on the second of those two issues, and it based around a journal article that I came across today that makes for interesting reading.

Effectively, each transport project that gets funded goes through a process of analysing its costs and benefits – to basically ensure that it’s worth spending money on. Then once these costs and benefits have been monetarised and aggregated, they are compared to the cost of the project to see how it stacks up. A cost benefit ratio of less than 1 means that the project will provide less benefit than its cost, while a ratio of above 1 means that its benefits are greater than its cost – effectively that it’s a project that might be worth doing. Of course, that makes everything sound quite simple, but the reality is quite different. The main question is “how do we calculate the costs and benefits?” Not only do we need to find a way to measure what the benefits would be (and often they’re quite broad benefits like enabling economic growth and so forth), but we also need to find a way of working out how to effectively apply a dollar value to each benefit. Furthermore, we don’t actually know what will happen in 10 or 20 years time once a project is complete (or not complete), so a cost-benefit analysis involves a lot of estimation and potential guess-work. We also need to look at costs of the project other than its simply construction and maintenance costs – like effects on carbon emissions, air pollution and other matters.

The aforementioned article does provide France as a case study, but most of the article stays quite broad in terms of an overview of what a cost-benefit analysis is, analysing how they’re undertaken, and (importantly) some criticisms of the process. At their simplest, a cost-benefit analysis (CBA) a way of ensuring you don’t waste money on something that won’t provide a decent return on that investment. The article provides further detail on the steps of how a CBA is undertaken:

The first phase consists of identifying the advantages and the disadvantages so that the effects to be taken into account can be measured. For each measurable effect, a physical measurement scale must first be specified. Then, an appropriate procedure for measuring the quantity of the effect to be evaluated must be conceived and applied to each competing project.

The second phase begins by making a list of the available appropriate procedures for assigning a monetary value to each physical unit identified in the first phase. These procedures maybe based on surveys known as stated preference or contingent valuation, emphasizing willingness to pay or receive. Since observing human behavior can reveal certain preferences, the procedures may also be based on the behavior exhibited by existing markets, allowing indirect estimates of the impact resulting from the benefits accrued or problems caused.

In the third phase of the monetary evaluation procedure,the future changes of a certain number of reference values considered in CBA calculations should be determined precisely. Therefore, it is necessary to plan for the future, an activity which leads to forecasting the reference values, particularly those concerning air pollution, noise or the implicit value of a human life.

The CBA that I am most familiar with is that undertaken for the Wateview Connection project. The version of the CBA that I include below relates to the previous full tunnel option, rather than the current partial surface/tunnel option, but still provides a useful guide for what a CBA looks like – and gives some hints as to how it was calculated.

The way these actual numbers were calculated is enormously complicated, involving very detailed traffic models, things called “discount rates” (the benefit of a project supposedly decreases by a certain amount each year) and the kind of strange thinking that separates travel time savings from congestion relief benefits. The Journal Article delves far deeper into those issues – so have a read of that if you’re interested.

Now we have some idea about how CBAs work, let’s have a look at some of the problems with them. My suspicion of time-savings benefits have been explained more fully in the past, but that is mainly a criticism of how one particular aspect of a CBA is undertaken, rather than a criticism of the concept of doing a CBA in general. Firstly, let’s have a look at whether a CBA really is the “objective analysis” that it appears to be trying to achieve. Because there are so many variables that go into a CBA, there ends up being a lot of assumption being made: what is the value of a human life when calculating safety costs/benefits? What is the value of time saved? Does it depend on a person’s profession? What is the value of noise pollution? Furthermore, because of the ‘discount rates’ applied, benefits that will be enjoyed a long time into the future are given far less value than those experienced in the near future. Is this really applicable given our need to improve inter-generational sustainability?

Once we have such a large number of assumptions, there can be a huge amount of debate that enters this supposedly neutral analysis. In my opinion, it seems as though the process of coming up with this “magic number” – the final cost-benefit ratio – is such a “grey” issue that the debate it is likely to generate should be open to public input. Yet, at the moment it seems as though the process is undertaken as a deep dark secret – under the impression that it’s a completely non-biased neutral process. The public simply doesn’t get the opportunity to question “is this project worth it?” – but instead during the consenting process can only argue about whether its environmental effects are acceptable or unacceptable.

The disconnection between the deep dark secret way that CBAs are conducted, and the general acceptance of public input into important decision making processes is a pretty big flaw in the way the current system works in my opinion. This disconnection is explored in the aforementioned journal article, and is the main concluding point of the whole article:

The evolution of the transport infrastructure decision-making context now obliges public decision-makers to worry about two new elements in the decision-making process: rationalization of public resource use and stakeholder acceptance of the choices made. Evaluating the socio-economic significance of projects using cost–benefit analysis techniques takes care of the first element. For this reason,we have described the principles of cost–benefit analysis and tried to show how it is performed in France. The second element has pushed the French legislature to progressively institutionalize the practices and tools that permit public debate.To illustrate this change, we presented the objectives and operating principles of the Public Inquiry Procedure and the National Commission for Public Debate.

In this paper, we support the thesis that cost–benefit analysis,as it is carried out today in France, is not compatible with relevant and constructive debate. In fact, in order to deal with the two concerns mentioned above, it is necessary to create a closer link between the processes of evaluation and debate. As we have attempted to show, this link can be improved relatively simply.In general,what is really at issue is the relationship between the requirements of rationality that the actors would like to see in every decision-making process and the requirements of legitimacy that are accrued through the collaborative process of public debate. These two requirements are not incompatible. In our opinion, the two can be reconciled by revamping the socio-economic evaluation tools and changing the way that economic evaluation is carried out.

To bring this back to the Waterview Connection, and make a bit more sense out of it – the current situation (and it is the same in both France and New Zealand) is that the project’s costs and benefits can be debated, but only in terms of its environmental effects through the Resource Management Act. There is no ability for me to publicly question, for example, exactly how NZTA came up with its $2.7 billion of time-savings benefits, or how they are different to the $690 million of congestion relief benefits, or what traffic model was used to determine the time-savings benefits, or whether the analysis of CO2 emissions has been done accurately.

I think it is quite obvious that working out a cost-benefit ratio for a project is not just a technical task for some faceless bureaucrat to undertake, but rather it is a contestable and debatable process – where a variety of different voices should be heard, and the assumptions that have inevitably been made throughout the CBA are probed and critiqued. After all, when there are enormous amounts of public funds being used, creating – as the journal article calls it – a sense of legitimacy through a collaborative process of public debate is likely, in my opinion, to lead to far better and more transparent decisions being made.

Share this