A couple of months ago I stated that if there were two things I could change about transport, it would be creating one pot of money for transport – and also to change the way that transport projects are analysed. This post focuses on the second of those two issues, and it based around a journal article that I came across today that makes for interesting reading.
Effectively, each transport project that gets funded goes through a process of analysing its costs and benefits – to basically ensure that it’s worth spending money on. Then once these costs and benefits have been monetarised and aggregated, they are compared to the cost of the project to see how it stacks up. A cost benefit ratio of less than 1 means that the project will provide less benefit than its cost, while a ratio of above 1 means that its benefits are greater than its cost – effectively that it’s a project that might be worth doing. Of course, that makes everything sound quite simple, but the reality is quite different. The main question is “how do we calculate the costs and benefits?” Not only do we need to find a way to measure what the benefits would be (and often they’re quite broad benefits like enabling economic growth and so forth), but we also need to find a way of working out how to effectively apply a dollar value to each benefit. Furthermore, we don’t actually know what will happen in 10 or 20 years time once a project is complete (or not complete), so a cost-benefit analysis involves a lot of estimation and potential guess-work. We also need to look at costs of the project other than its simply construction and maintenance costs – like effects on carbon emissions, air pollution and other matters.
The aforementioned article does provide France as a case study, but most of the article stays quite broad in terms of an overview of what a cost-benefit analysis is, analysing how they’re undertaken, and (importantly) some criticisms of the process. At their simplest, a cost-benefit analysis (CBA) a way of ensuring you don’t waste money on something that won’t provide a decent return on that investment. The article provides further detail on the steps of how a CBA is undertaken:
The first phase consists of identifying the advantages and the disadvantages so that the effects to be taken into account can be measured. For each measurable effect, a physical measurement scale must first be specified. Then, an appropriate procedure for measuring the quantity of the effect to be evaluated must be conceived and applied to each competing project.
The second phase begins by making a list of the available appropriate procedures for assigning a monetary value to each physical unit identified in the first phase. These procedures maybe based on surveys known as stated preference or contingent valuation, emphasizing willingness to pay or receive. Since observing human behavior can reveal certain preferences, the procedures may also be based on the behavior exhibited by existing markets, allowing indirect estimates of the impact resulting from the benefits accrued or problems caused.
In the third phase of the monetary evaluation procedure,the future changes of a certain number of reference values considered in CBA calculations should be determined precisely. Therefore, it is necessary to plan for the future, an activity which leads to forecasting the reference values, particularly those concerning air pollution, noise or the implicit value of a human life.
The CBA that I am most familiar with is that undertaken for the Wateview Connection project. The version of the CBA that I include below relates to the previous full tunnel option, rather than the current partial surface/tunnel option, but still provides a useful guide for what a CBA looks like – and gives some hints as to how it was calculated.
The way these actual numbers were calculated is enormously complicated, involving very detailed traffic models, things called “discount rates” (the benefit of a project supposedly decreases by a certain amount each year) and the kind of strange thinking that separates travel time savings from congestion relief benefits. The Journal Article delves far deeper into those issues – so have a read of that if you’re interested.
Now we have some idea about how CBAs work, let’s have a look at some of the problems with them. My suspicion of time-savings benefits have been explained more fully in the past, but that is mainly a criticism of how one particular aspect of a CBA is undertaken, rather than a criticism of the concept of doing a CBA in general. Firstly, let’s have a look at whether a CBA really is the “objective analysis” that it appears to be trying to achieve. Because there are so many variables that go into a CBA, there ends up being a lot of assumption being made: what is the value of a human life when calculating safety costs/benefits? What is the value of time saved? Does it depend on a person’s profession? What is the value of noise pollution? Furthermore, because of the ‘discount rates’ applied, benefits that will be enjoyed a long time into the future are given far less value than those experienced in the near future. Is this really applicable given our need to improve inter-generational sustainability?
Once we have such a large number of assumptions, there can be a huge amount of debate that enters this supposedly neutral analysis. In my opinion, it seems as though the process of coming up with this “magic number” – the final cost-benefit ratio – is such a “grey” issue that the debate it is likely to generate should be open to public input. Yet, at the moment it seems as though the process is undertaken as a deep dark secret – under the impression that it’s a completely non-biased neutral process. The public simply doesn’t get the opportunity to question “is this project worth it?” – but instead during the consenting process can only argue about whether its environmental effects are acceptable or unacceptable.
The disconnection between the deep dark secret way that CBAs are conducted, and the general acceptance of public input into important decision making processes is a pretty big flaw in the way the current system works in my opinion. This disconnection is explored in the aforementioned journal article, and is the main concluding point of the whole article:
The evolution of the transport infrastructure decision-making context now obliges public decision-makers to worry about two new elements in the decision-making process: rationalization of public resource use and stakeholder acceptance of the choices made. Evaluating the socio-economic significance of projects using cost–benefit analysis techniques takes care of the first element. For this reason,we have described the principles of cost–benefit analysis and tried to show how it is performed in France. The second element has pushed the French legislature to progressively institutionalize the practices and tools that permit public debate.To illustrate this change, we presented the objectives and operating principles of the Public Inquiry Procedure and the National Commission for Public Debate.
In this paper, we support the thesis that cost–benefit analysis,as it is carried out today in France, is not compatible with relevant and constructive debate. In fact, in order to deal with the two concerns mentioned above, it is necessary to create a closer link between the processes of evaluation and debate. As we have attempted to show, this link can be improved relatively simply.In general,what is really at issue is the relationship between the requirements of rationality that the actors would like to see in every decision-making process and the requirements of legitimacy that are accrued through the collaborative process of public debate. These two requirements are not incompatible. In our opinion, the two can be reconciled by revamping the socio-economic evaluation tools and changing the way that economic evaluation is carried out.
To bring this back to the Waterview Connection, and make a bit more sense out of it – the current situation (and it is the same in both France and New Zealand) is that the project’s costs and benefits can be debated, but only in terms of its environmental effects through the Resource Management Act. There is no ability for me to publicly question, for example, exactly how NZTA came up with its $2.7 billion of time-savings benefits, or how they are different to the $690 million of congestion relief benefits, or what traffic model was used to determine the time-savings benefits, or whether the analysis of CO2 emissions has been done accurately.
I think it is quite obvious that working out a cost-benefit ratio for a project is not just a technical task for some faceless bureaucrat to undertake, but rather it is a contestable and debatable process – where a variety of different voices should be heard, and the assumptions that have inevitably been made throughout the CBA are probed and critiqued. After all, when there are enormous amounts of public funds being used, creating – as the journal article calls it – a sense of legitimacy through a collaborative process of public debate is likely, in my opinion, to lead to far better and more transparent decisions being made.
Processing...
Another interesting article you have. I would point out that one key part of a cost benefit analysis should be repeatability – ie that the same set of criteria are applied to each project. And again, the difficulty comes into how much do we value a life. In one sense, lives are priceless; in another they are worthless (in that they don’t impose any actual cost to the road operator). Not saying I agree with either of these, but safety works need these calculations.
Keep on pushing these ideas and challenging this though – it is interesting. I would love to see a full cost benefit of a number of things; the second harbour bridge, the proposed CBD rail tunnel, widening of the northwestern motorway. What would also be interesting is to evaluate the actual costs of operation – ie motorways for their entire real estate, cost of presence through pollution, train lines with cost of operation, etc. All interesting but a fulltime job to even scratch the surface!
NZP
A further point, NZTA seem to be publishing the basis of their cost benefit analysis methods – far from a ‘deep dark secret’.
http://www.landtransport.govt.nz/funding/manuals.html#eem1
http://www.landtransport.govt.nz/funding/manuals.html#eem2
Wow 689 pages for volume one alone. That’s going to take some time for me to trawl through there. I’m not so much critiquing that the plan is a secret (although I was not aware it was on the NZTA website), but that the nuts and bolts of the analysis aren’t open for public debate.
Yes, I think NZ’s method is severely flawed… I think England learned their lesson after the “Roads to Prosperity” and changed their BCR process, maybe we will after the “Roads of National Significance”… Both are of course, “Roads to Congestion”…
It will be interesting to see how the “RoNS” issue develops over the next few months. There’s going to be some serious questions raised once there is a cost-benefit analysis of some of them, particularly the Puhoi-Wellsford road.
It has never been a deep dark secret. NZTA and LTNZ and Transfund before have undertaken regular reviews of the economic evaluation framework. These have always been publicised, but frankly unless you are an economist or have a good enough understanding of trnaposrt economics to understand the strategic impact of various forms of calculation, you probably cannot get to grips with how it works. The fact you don’t understand what a discount rate is indicates you’re no economist. The public have negligible input into any of this, except selection of options that have little difference between them on economic grounds. The reason is obvious, because of significant capture of any consultation by lobbyists and NIMBYism. The loudest shout isn’t necessarily the widest range of people or impacts.
Cost-benefit analysis has TWO purposes. The first is prioritisation. It allows projects to be ranked according to value, so that regardless of the BCR 1 point, at least the net impact is well quantified. Prioritisation is something rarely discussed, but is obviously critical. The second is determining whether projects have value at all. That is the BCR 1:1 or less scenario. Treasury told me years ago that a BCR of 2.5:1 was a more sensible threshold for most projects, but that was under tighter budgetary conditions.
The key inputs are standardised values around time, elasticities of demand and forecasts of future growth and economic conditions. Following that come evaluation periods (which frankly should be based on the depreciated life of whatever is being bought, but becomes standardised to determine prioritisation), sensitivity testing and then hopefully optimism bias – every proponent of a project underestimates costs and overestimates demand, and this propensity needs to be taken into account.
CBA is a tool, a tool to help the allocation of scarce resources from a surplus of funds beyond that needed to maintain the road networks. In the past it has attempted to replicate what road users want, by buying projects that have the highest value to them on the basis of values that reflect those (e.g. travel time is highly valued, as it journey ambience and high safety risk).
“There is no ability for me to publicly question, for example, exactly how NZTA came up with its $2.7 billion of time-savings benefits, or how they are different to the $690 million of congestion relief benefits, or what traffic model was used to determine the time-savings benefits, or whether the analysis of CO2 emissions has been done accurately” Nonsense you can OIA the lot of this, and if you think NZTA has breached its own procedures or indeed the law, write directly to the CEO or Chair. If you have enough support, you can take it for judicial review for breach of administrative procedures. It is relatively easy to find out how all of this was calculated, but if the issue is around policy decisions, then NZTA will engage with you on this. I suggest you try.
By the way CBA’s do sometimes get peer reviewed and are sometimes audited post project completion to see if the appraisal was right about cost, demand and impacts.
Jeremy Harris: The UK has a far more politically manipulated process, and does not have a dedicated funding agency, and the BCR appraisal period is a ridiculous 70 years (and now includes all sorts of “wider benefits” that are increasingly qualitative). Roads to Prosperity and the subsequent No Roads policy, and backtrack on that in the UK have little to do with appraisal. Indeed approval of large projects doesn’t either. The Channel Tunnel Rail Link only had a barely positive BCR because of claimed regeneration benefits in Ashford and Ebbsfleet which have simply not materialised. In other words, it would have been better value to put the money into a foreign government bond.
You can get BCRs for every road project and every capital public transport project funded by NZTA and its predecessors. Transfund used to publish BCRs in the National Roads Programme but Labour expressly had this removed from the National Land Transport Programmes, because it would show up high BCR projects being delayed and low BCR projects being advanced ahead.
Puhoi-Wellsford might have publicity, but there are more than a few other marginal projects. For example, two stretches of the Waikato Expressway are lucky to have BCRs of 1. Those are the Huntly Bypass and the Hamilton East segment. All the others have BCRs over 2 IIRC. In other words, it is worth having the expressway completed north of Huntly, and from Taupiri to Te Rapa, and from Hamilton to Cambridge (including bypass), but those segments are either too expensive (Huntly) or there is insufficient demand (Hamilton East).
Thanks for those useful insights Liberty. Indeed, I am a Town Planner and not an economist, hence my experience is in terms of the resource consenting process – which for large projects is open to public input and critique. I don’t buy the whole ‘you have to be an expert to deserve a say’ in terms of project prioritisation. As long as you have a robust framework and impartial decision makers then I think adding debate can only be a good thing. It might add a few months to the process, but when we are talking about spending billions of dollars does it not make more sense to ensure you make the right decisions,
rather than fast decisions?
If the public was more involved in the process then I think it would be more legitimate and there would be greater buy-in. You say yourself that many low BCR projects have been unduly prioritised. This might reduce that.
I don’t see 70 years as a ridiculous time period, our railway corridors are over 100 years old, does any rational person doubt the CMJ is going to be there in 70 years..?
Indeed. In most large cities around the world there is still huge benefit gained from rail investment around a century ago.
A 2009 paper on the website of GAMUT, the Australasian Centre for the Governance and Management of Urban Transport (Melbourne University), argues that predicted travel time savings for Melbourne’s motorways have not been realised, most probably because of the induced-traffic phenomenon identified by British planners: “In sum, these results suggest that the core of travel times savings benefits, which is an increase in average travel speeds, has not eventuated in Melbourne’s urban road network during the years under review. Indeed, based on the evidence presented and analysed in this paper, one could be led to the conclusion that investments in Melbourne’s urban road network have resulted in more time being used by Melbourne’s motorists rather than less time. On the basis of the assumptions conventionally used to justify road building , major road infrastructure initiatives have resulted in net economic disbenefits…This conclusion is most concerning, given the very substantial amounts of public funds that have been directed at Melbourne’s urban road network over the last several decades, coupled with the dominance of claimed travel time savings in the economic cost-benefit justifications that have preceded the approval of such very large scale additions to Melbourne’s freeway stock…This analysis clearly raises very real doubts about the validity of using the projected value of travel time savings as the basis for estimating the social economic benefits of building new road projects, at least within established and complex urban road networks. Even if this validity were accepted, there is a very strong need for far more rigorous and empirically based methods to be used when projecting the quantum of travel time most likely to result from the road project being evaluated at the time.” …. See http://www.abp.unimelb.edu.au/gamut/pdf/have-all-the-time-savings-been-achieved.pdf .
Indeed Chris, I did a whole post on that study: http://greaterakl.wpengine.com/2009/09/30/time-savings-benefits-real-or-not/
There’s also a very interesting article by UK academic David Metz too: http://greaterakl.wpengine.com/wp-content/uploads/2009/10/Metz-2008.pdf
Jeremy Harris: The problem with 70 years is what is the depreciation period of what you are building. With the exception of land and earthworks, roads are completely rebuilt twice in that period, railway lines at least once, more if heavily used. It needs to reflect the depreciated life of the asset concerned, or that plus capital needed to keep it usable over that time. My main problem with 70 years is that any forecasting becomes rather nonsensical over such a period. It assumes no technological obsolescence, it assume current trade and urban/rural growth patterns. For example, anyone evaluating a project in 1949 for today would have not even considered how containerisation would change freight movements, the enormous shift of domestic travel from rail to car and increasing aviation, the end of international scheduled passenger liner services, the end of rail as a viable mode for most freight movements under 250km, the emergence of fleet management and road pricing technologies. Assets with very long lives better be worth it in the end, New Zealand has had more than a few failures on that front, like Wellington’s Overseas Passenger Terminal for shipping opened in the mid 1960s, the Wahine (III) and Rangitira (II) for the Lyttelton ferry run, the Silverstar…
Fair enough Liberty you changed my mind…