In several recent posts I’ve taken a look at people’s revealed preferences for roads (nobody’s willing to pay directly for them) and public transport, walking, and cycling (people are queuing up to get on the train). In those posts, I’ve argued that observing how people vote with their feet (or their wallets) can teach us a lot about demand for different travel modes.

Rail is now growing too fast to be un-fit for survival.

But as any economist knows, markets have two sides to them: demand and supply. As transport infrastructure has a lot of “public good” characteristics, it tends to be provided by government agencies such as Auckland Transport and the New Zealand Transport Agency. (These agencies wouldn’t say no if a private company turned up and offered to build a new motorway at no cost to them… but that’s not going to happen any time soon due to the fact that most recent private toll roads have failed financially.)

As a result, we have to consider how transport agencies make decisions about what to supply to the market. I’ve written a few posts on the basics of cost-benefit analysis, which is one of the tools that they use to decide which projects to build.

But is cost-benefit analysis robust, or are the results systematically biased in a certain direction? Thinking about this question led me to re-read one of my favourite papers on infrastructure costings (don’t laugh!): Bent Flyvbjerg’s “Survival of the Un-fittest: Why the Worst Infrastructure Gets Built – and What We Can do About It” (fulltext pdf). Flyvbjerg takes an empirical look at hundreds of major infrastructure projects around the world, finding that cost overruns are all-pervasive:

  • 9 out of 10 projects have cost overrun.
  • Overrun is found across the 20 nations and 5 continents covered by the study.
  • Overrun is constant for the 70-year period covered by the study, cost estimates have not improved over time.

In addition, benefits are systematically overestimated in ex-ante evaluations. The result is that a number of bad projects get built on the back of over-optimistic business cases. Flyvbjerg attributes this to “cognitive and political biases such as optimism bias and strategic misrepresentation”. (This is a polite way of saying “lying about the project to ensure that it gets built.”)

So how do New Zealand’s transport agencies stack up against Flyvbjerg’s analysis? Fortunately, we’ve got some empirical data to investigate this question with. Between 2009 and 2012, NZTA conducted and published a number of post-implementation reviews of (mainly) road projects that it funded in part or fully. Matt did an excellent job summarising the data in a post last year.

While the projects aren’t necessarily representative of all road projects, they do run the gamut from small pavement upgrades to multimillion state highway expansions. NZTA provided data comparing ex-ante and ex-post evaluations of costs and benefits for 69 projects in total. I subjected the data to some basic statistical analysis, finding that:

  • The average project had a cost overrun of 34% – a difference that was found to be highly statistically significant, meaning that there is a less than 1% probability that the observed difference happened by chance.
  • The average project had actual benefits that were 28% lower than expected – although as this difference was not statistically significant we can’t determine whether it simply reflects random chance.

In other words, NZTA and regional transport agencies seem to have had some issues accurately costing road projects. And the errors they are making are not random – they have systematically underestimated costs. This can be seen really clearly if we graph the data in histogram format.

Here’s the data on construction cost overruns, in percentage terms. The size of the bars represents the number of projects. Bars to the right of the black line indicate projects where costs were higher than expected. As you can see, costs were higher than expected for the vast majority of projects – sometimes to a quite significant degree (i.e. over 100% more expensive than planned).

Road cost overrun chart

And here’s a similar chart for benefit overruns/underruns. This shows that although estimates of benefits have in some cases been wrong by a quite large amount, most of the errors are clustered closer to the zero line. This shows that while NZTA or transport agencies often miss the mark on their estimates of benefits, the errors are sometimes positive and sometimes negative. In other words, optimism bias seems to be less pervasive when estimating benefits than when estimating costs.

Road benefit overruns chart

This data has (or should have) important implications for the way we plan and fund transport projects. It suggests that it’s necessary to be much more conservative when estimating the costs and benefits of road projects. This is especially important in light of the fact that NZTA’s funding is being devoted in large part to major motorway projects – the kind of “megaprojects” that Flybjerg identifies as posing the greatest risks for good project evaluation.

Unfortunately, NZTA stopped publishing post-implementation reviews in 2012, so it’s impossible to say whether agencies have used this data to refine their cost estimates. I hope they have, but there are indications that optimism bias is still running rampant. Take, for example, NZTA’s long-term forecasts of road traffic and public transport patronage, which blithely disregard the market realities. Or, more concretely, there’s the strange case of the Additional Waitemata Harbour Crossing traffic forecasts, which Matt picked up on a few years ago.

A 2010 business case for the AWHC, which would be New Zealand’s most expensive infrastructure project of all time, found that the project’s benefit-cost ratio was a mere 0.4 to 0.6. (Indicating that it costs about twice as much as it returns in benefits.) But, as it turns out, this figure was based on traffic modelling that overestimated actual traffic across the bridge in 2008 by almost 10% – in spite of the fact that the actual data was available at that point. That’s some serious optimism bias right there…

Ak Harbour Bridge Traffic volumes
Auckland Harbour Bridge Traffic volumes (actual and forecast)

Finally, it’s also worth noting that Flyvbjerg finds that cost overruns (and benefit underruns) tend to be a more serious issue for rail projects than for road projects, especially in the United States. Unfortunately, we simply haven’t completed enough rail projects to robustly evaluate whether the same holds true in New Zealand. However, there are some signs that recent public transport infrastructure projects have outperformed their business cases – as seen in NZTA’s post-implementation review of the Northern Busway and booming ridership at Britomart.

Share this


  1. Great post. It’s important that NZTA learn from what comes out of the post implementation reviews as they are the one true check of the “complicated thumb suck” process used to spend billions of public money.

    The AWHC data is shocking. I often feel there’s something incredibly dodgy about that project going on behind the scenes.

  2. Why are NZTA no longer conducting post implementation reviews? Or are they just not being published? Could this be linked to the glorious RoNS, and widespread failure of traffic to turn up in volumes predicted?

  3. Surely a recent case in point for this issue has to be the newly opened Te Horata road under the EP highway and beside the Panmure station part of AMETI.

    I’ve driven on and over it several times since it opened for traffic on 2 November (some 2 weeks now), and I’ve yet to see a single one of the 2400 trucks a day that are supposed to use it, actually, you, know, using it.
    Doesn’t bode well for one of the project’s main benefits.

    Perhaps the restriction on dangerous goods in the “tunnel” section is deterring the trucks in droves.
    I recall, this particular NZTA/AT wonder-project was a lot over budget (and time) before it openeded.
    The road was supposed to be open mid 2014, then it kept slipping, further back, to November 2014, and yet the footpaths are still not completed given all that extra time.

    Cars mind you are using it, just not trucks. Another case where BCRs are all about the Benefits, while ignoring the costs and Ratio part of the BCR equation?

    Agreed this road will allow the Panmure roundabout to be redeveloped, but it has come with a huge price tag attached.

  4. Peter,
    If you were to focus on all the projects with higher than expected costs (10% overrun or higher), and then graph their actual benefits distribution like you did for the second graph, what does the second chart look like then?
    [allowing for the fact that some over or under run by 10% is probably accepted practice, so we’ll focus only on those that overrun costs by 10%].

    I’d suspect that the underrun in benefits for these “expensive” projects get worse and worse as the overrun cost spirals up.
    I’d also expect that many of the large “benefits underruns are the side-effects of big mega projects not medium of smaller projects under running their benefits.

    I’d also suspect that for those under running by more than 10% projects the benefits are also curtailed strongly too, suggesting the savings are coming from not delivering the benefits it promised rather than on “wise spending” (or poor estimating of the actual costs).

    And in that case, the way to get best bang for buck is to stop the big risky projects before they start and also stop those underspending projects that cut the benefits out.

    1. Interesting question! I took a brief look at it. There were 38 projects where costs overran by more than 10%. However, those projects weren’t especially prone to have benefit underruns. If anything, they were _less_ likely to experience benefit underruns than projects where costs didn’t overrun to such a large degree.

      One interpretation of this, I suppose, is that cost overruns are related to the fact that transport agencies or contractors have identified additional things that have to happen in order for promised benefits to actually be realised.

  5. A great post and would appear to indicate that NZTA is not performing to the standard taxpayers can reasonably expect. They are either out of control or their political masters are incompetent. It seems to me that a total reform of the Ministry of Transport is needed including an overhaul of funding mechanisms and funding allocations. Tweaking it would not be enough.
    An opportunity for the new Minister to really make his mark.

    1. Crap, you and peter have both confused the tendered price vs the project cost estimate, which you have no idea of. That is the important bit. If the nzta are experianced and smart in this field they will set up the project award process to drive down the price, then argue the variations with the construction team to deliver best value for the government.

    2. Hi Warren – I wouldn’t say that this data shows that NZTA or regional transport agencies are especially bad by world standards. Flyvbjerg’s research shows that cost overruns are rampant in the infrastructure sector almost everywhere.

      That being said, there is a case to improve discipline around cost-benefit analysis and project costings. I think NZTA deserves some credit for having done these post-implementation reviews, as they provide a useful evidence base for improving project costings. But they need to go a step further and actually make some changes in response!

      With regards to Pete and Tracy’s comment: NZTA’s published data suggests that ex-ante cost/benefit estimates reflect the figures used at the point at which the projects were approved. They reflect the figures that NZTA used to decide whether to fund a project. In other words, I have compared actual build costs with NZTA’s original cost estimate, rather than with the tendered price.

  6. Cost over-runs are not restricted to large infrastructure projects. Anyone built or renovated a house recently? If it’s one of the Neil Homes cookie cutter variety it should be bang on budget, but most NZ homes aren’t and nor are infrastructure projects. Yes, there are many common components and features between one project and the next but there will always be unknowns … and the weather. All this tells me is that post implementation reviews must be mandatory and the “what we’ll do differently on the next project” must also be published. Nothing can be done to change the past and all we can ever hope to do is influence future behaviours to get better outcomes next time. The source, interpretation and use of benefit estimates should be questioned in the same analysis and be part of that intention to do better next time.
    However, when the intended benefit of an infrastructure project is to curry favour with one interest group over another no amount of a priori or post priori of benefit analysis will do a darned bit of good. And I think we might be seeing some of that hence the end of NZTA post analysis. I’m happy to be proven wrong but up until then I am sceptical of their motives.

  7. It should be noted however that there is, if anything, a ‘pessimism bias’ when it comes to Public Transport projects. Our institutions consistently get ridership numbers wrong on the low side and often by considerable amounts. They did with Britomart and the Northern Busway. What it seems to be is that their expectations, training, and models are all out of date; expecting not only the strong traffic growth from last century but also the weak Transit demand.

  8. Yes we are building the wrong things. We should reprioritise by space the roading assets/corridors we do have and have already invested in. If Auckland as transport experts were all working together we could be maximising all routes for public transport as No1 then the other modes.Get the network up with what we have then and only then look at road widening. Buses held up in cues is this really smart thinking. No. If the mayor wanted to fix things give public transport first say over the entire existing network including motorways. Guarentee our roading Capex would reduce by 75% plus and no extra $12b. Focus on this first then look at change in behaviour. But currently it is NZTA with their turf , budgets,car modelling and council to afraid to even upset a few shop owners. If the major sold this change as a global network reprioritising reducing roading costs overall. If we did this a network wide game changer and then focus on rail/PTand roading in growth areas only.

Leave a Reply

Your email address will not be published. Required fields are marked *