One of the most important parts of any project is to review it after completion to see what went right, what went wrong and to see if the project did what it was expected to. This is especially important with transport projects due to the often massive costs involved in new or upgraded infrastructure. Thanks to an OIA request, a year I obtained the post implementation review (PIR) for the Northern Busway while in October another request showed that for many of the roading projects within Auckland had been overstated. I thought I would do the same thing again to see if any more reviews had been completed since then. However to their credit, the NZTA have now started publishing the details of all PIRs online and they made for interesting reading.
Rather than wade through and pick out a couple of examples, I have instead collated the results into a spread sheet to make it easier to compare. Some of the projects are from local bodies while some are from the NZTA. There are also a couple of relating to public transport or cycleways. I have highlighted in red or green if the project performed better or worse than expected.
While each review was unique, reading through the comments very similar recurring themes turned up.
- Roading improvements often didn’t have the safety benefits promised and in some cases made the road less safe due to inducing more traffic and/or higher speeds.
- Traffic volumes didn’t increase as expected.
- Documentation including the project assessment was often not complete or missed out crucial information.
- The original assessment didn’t follow the evaluation guidelines and in one case the local authority made up its own criteria that among other things, ignored on-going maintenance costs.
What is important to note is that despite performing worse than expected, many of the projects were still quite positive and worth building but had they been assessed properly they might not have been given the priority that they were.
I will be keeping an eye on future PIRs that are done to see if the standard starts to improve.
Processing...
Could you post the source spreadsheet (or put in a public Google doc?). It would be interesting to look a the statistics here because just eye-balling it you have to wonder if the modelling is achieving variability less then (say) 3-5 BCR.
It is only basic and I just took the information out of the NZTA reviews linked to above but here it is
https://docs.google.com/file/d/0B0f0ymGg1y-8YWlyRVpTdEJCY0E/edit?usp=sharing
Here is the BCR error frequency plotted:
http://i.imgur.com/BvZc98dl.png
So, yeah, negative skew around an average absolute error of 3.9 and a standard deviation of 4.6. Feel free to correct my rudimentary statistics, but I you could crudely say that modelled BCR’s are 95% accurate to within +/- 10.
Quick question though, how are BCRs calculated from the costs and benefices?
A more useful data point would be the actual benefits vs projected costs. I.e: did the project provide the benefits anticipated with the costs planned?
My thoughts too – is there some discounting calcs going on behind the scenes? Because “benefits divided by costs” generally isn’t coming up with the BCR value; in some cases quite a lot out.
Yes there are things going on in the background to take into account issues like when the costs and benefits occur. Some of the notes point out that the changes only come about because the sequencing of parts of a project may have changed and so impacts on when the benefits are achieved.
All projects over $10M performed worse than expected, which suggests expectations for large projects need to be adjusted down (not thinking of any upcoming ones in particular. Oh sorry- RONS have a “Get out of BCR free” card, as they’re being done because Nats promised them, not because they stack up financially).
(I’m ignoring Whitford-Maraetai Rd in the table, as pre and post of “2607” suggests someone got paid $10K to copy & paste).
Thank you Matt. That ‘BCR Post’ column is shocking.
Wow what happened with that Ashburton business park? $990,000 of spending to create a $52m disbenefit.
Or the Windy Ridge passing lane extension which also shows a disbenefit? Maybe these routed the road through a bombing range or something?
The NZTA basically took a development risk. They built the the project on the expectation that a whole lot of businesses would set up in this business park and at the time of the review, none had.
Windy ridge was disbenefits due to higher accident costs.
BTW the details of the review for each of these projects are in the links above including what factors led to the results being different.
http://www.nzta.govt.nz/planning/monitoring/audits/pir.html
Not many cycle projects on that list, then again not surprising.
Build them and they would come:
http://www.nzherald.co.nz/nz/news/article.cfm?c_id=1&objectid=10888174
The electric bike concept seems to be slowly “taking off” in hilly cities e.g.
http://www.scotsman.com/edinburgh-evening-news/transport/edinburgh-mulls-electric-bike-hire-scheme-1-2946013
http://green.blogs.nytimes.com/2012/02/06/a-bay-area-experiment-in-electric-bike-sharing/
Maybe time for some questions in the House on the sea of red to the Minister of Roads
The Opposition seem to miss the obvious – not sure what their research teams do all day….?
The Greens do a lot of research and ask sharp questions – then Joyce / Brownlee / Key answer with a variation of the “You freaks want us all to go back to the horse & cart days, don’t you?” not-really-an-answer, and that’s it for them.
Matt could you sum the columns and calculate the percentage difference between pre and post benefits and costs? I’m interested to know whether most of the variance come from 1) costs being higher than expected or 2) benefits being lower than forecast or 3) a combination of both.
From eye-balling the figures it’s somewhat hard to tell …
In total costs were 23% higher than predicted while benefits were 13% lower.
Optimism bias?
Methodical bias?
Hope we see some follow-up questions about Windy Ridge. Wonder who missed that potential impact during project planning?
Some the biggest projects date back to the late 1990s and early 2000s (Otira Viaduct/ Fairfield Motorway). What the planners could not reasonably anticipate was that China’s economy was about to send fuel and construction prices skyrocketing. It’s hardly surprising that costs went up by 25% and traffic volume related benefits went down by 13%. The results might be different if the projects are divided into small and large projects as the smaller projects should have less construction price inflation effects due to shorter planning/construction timeframes.
It was pretty easy to predict that.
A constrained supply of oil, growing instability in the middle east, and massive growth in the BRICs was set by 1990