One of the criticisms that we have had of the NZTA in recent years has been that there they don’t appear to go back and check if the what they predicted to happen turned out to be accurate. This is pretty important as they are spending huge amounts of our money on various projects and as we showed last week, it appears that sometimes they aren’t even using real data in their business cases. I reported on the review of the northern Busway back in June so I decided to ask them (via an Official Information Act request) for any other post implementation reviews that they have done in the last 5 years for the Auckland region. Here is what they provided me, first up the letter that came with the response:
Five post implementation reviews of state highway projects in the Auckland region have been carried out over the last five years. These are as follows:
- State Highway 1 Stafford to Esmonde bus priority lane
- Newton Road to Western Springs auxiliary lane
- Puhinui intersection grade separation
- Northern Busway
- Greenlane East amalgamated improvements
Please refer to Attachment G for summaries of these reviews.
While these reviews have identified ways that we can improve our project planning and management practices, they are not a statistically significant sample. For this reason, we consider their results and findings to be indicative rather than conclusive.
The NZTA has two initiatives underway to better measure the outcomes from our investments. These are:
- Approved Organisations (including the NZTA’s Highways and Network Operations group) will be required to formally agree performance measures at the time of funding approval and to report back at agreed dates. In time, this will provide a much more statistically robust picture of the performance of the improvements programme overall. This initiative is in the final stages of development and is to be applied to funding approvals from 1 July 2012. We recognise that it will take time to gather the post-implementation information and we will need to continue with our current programme of post implementation reviews in the meantime.
- We are reviewing our post implementation review methodology. The intention is that the new methodology will focus on the underlying benefits of a project rather than being based on a recalculation of the benefit-cost ratio. The new methodology will investigate and explain variancesin both cost and benefit achievements in greater detail so that lessons can be learned and applied to future decision making.
You will note that some of the reviews are for projects that were started some years ago. Where applicable, one of the pieces of information we collect before starting projects is the crash history of the section of road over the previous five years. To allow us to rate how effective the project is in reducing the number of crashes, we need to compare the earlier crash history data with that collected at least three years after the project is completed.
What I found immediately interesting was how few projects were on the list considering the amount of NZTA projects that have gone on in recent years, e.g. why has no PIR been done on the section of SH18 from Greenhithe to Albany Highway, or the extension of SH20 to Maioro St. I’m sure there have been quite a few other projects finished within or close to that timeframe. I was also surprised by just how long the explaination was but that perhaps made a bit more sense after reading the attached document. It was really just a summary of the outcomes but there are certainly very concerning comments from the auditors. Here are the road based ones.
Newton Road to Western Springs auxiliary lane
Greenlane East amalgamated improvements
Puhinui intersection grade separation
So while I agree this isn’t a statistically significant sample, it certainly isn’t a good look. It hints that there are certain elements within the organisation that are so determined to build roads they will ignore economic advice and funding conditions to get things done. It really makes me wonder what else has been going on within the organisation and will the big motorway projects like Manukau and Hobsonville etc. get similar results. We certainly shouldn’t be thinking of embarking on building another harbour crossing for a cost of $5 billion if we can’t get projects like these correct.
Here are the comments for the other two projects:
State Highway 1 Stafford to Esmonde bus priority lane
Northern Busway
Processing...
Anyone know how the BCRs are calculated? Dividing the benefit dollar amount by the cost dollar amount gives a slightly lower number than the quoted BCR.
The most obvious one is Greenlane post-implementation (Cost $24.7m higher than benefit $22.3m, but BCR quoted as a positive 1.1).
The costs haven’t been changed to reflect their “net present value” which applies a discount rate. Net present value cost is always lower than total cost, therefore bringing up the BCR.
Are you sure? I would have thought that the benefits/costs are quoted in NPV figures. I hope you’re right!
The costs are stated.
The benefits are stated
The BCR (benefit-cost ration) is stated
it’s either a failure of arithmetic or a failure of semantics ie 2 different (unstated) definitions within the same document. Either way it’s unprofessional.
Either that or NZTA can’t do basic division….
None of them add up so you could be correct.
I would be very surprised if the benefits and costs were not quoted in NPV terms in some way.
You’ve identified exactly what I was going to comment on.
The “post-implementation estimate” of the BCR for the “Greenlane East amalgamated improvements” project appears to have been calculated incorrectly. BCR = benefits/costs = $29.4 million / $34.7 million ~= 0.9.
It seems that NZTA, or their auditors, have accidentally inversed the calculation, i.e. $34.7/$29.4 = 1.1. If that is the case then not only does this process call into doubt some of the projects, but also the robustness of the PIR process itself! Dear oh dear.
Laughable that the northern busway could really have been expected to have such a pathetic BCR.
Part of that is because the in the benefit calculations, the time of PT users is valued less importantly than that of drivers. There is a full copy of that post implementation review linked to in the post above.
That looks like the one they were most accurate on??
Or if you actually read it you would find that PT users are valued less than car drivers as I mentioned.
Are these BCRs being calculated on an individual project basis worthwhile? Should the BCRs not be completed (if at all) on the overall network scheme (for the region) rather than on an ad-hoc project by project basis?
Also do we need the NZTA within the Auckland Region? The previous territorial authority & local councils have been amalgamated for efficiency & to remove potential conflicts, perhaps amalgamating the NZTA’s Auckland based responsibilities & associated revenue – with Auckland Transport would also provide something similar?
Andyt, yes and Auckland just receive a proportionate share of the NLTF to invest as it sees best?Good grief! Outrageous! That’s democracy, we can’t have that sort of crazy thinking. How could we prevent investment in Public Transport then, make sure all the overbuilt and uneconomic motorways that have been promised to the road lobby happen?
This really is appalling. The most galling thing about all this is that NZTA can stand up in the Environment Court and pronounce that the Western Ring Route will produce benefits that outweigh the costs, yet there isn’t a shred of evidence that these will be achieved, and the Environment Court isn’t interested to know if other roading projects have met any success criteria. It is unbelievable that projects like SH20/SH1 (remember the queues of traffic that have led to yet another road widening project?), the Manukau Harbour crossing and Grafton Gully haven’t been subject to a post-implementation review. It really is scandalous that there is no accountability here.
I’ve found that the mentality in New Zealand is that we tend to put issues of accountability in the “too hard” basket, as it raises too many awkward questions for the organisations concerned.
ok, so with 2 out of 3 motorway projects the cost almost doubled and the benefits halved. With the bus projects, the costs were almost on target and the benefits were also consistent (if low because of the weird methodology used in the EEM). Hmmm, one shudders to think what would happen if they evaluated the really big motorway projects? Really, you’d think the Minister should order reviews of them all to check this is not happening all the time…
And on the third roading project the reassessed benefits were less than predicted.
Looks like the Govt couldn’t give a toss about robust environmental analysis either
http://www.scoop.co.nz/stories/PA1210/S00477/national-scraps-crucial-environmental-report.htm
just wanted to add – good on you Matt for getting this info. Had heard rumblings about this for some time but never seen it OIAed before.
Hahaha I am laughing so the tears dont come.
I am thinking of writing a book based on Kafka’s “The Castle” about NZ’s transport policies. However, perhaps even Kafka couldnt believe how FUBAR it is. Makes the Austro-Hungarian administration and policies look quite sane.
Shshsh ! Best not to let the North Shore Busway evaluation project report come to the attention of the Minister until after it’s published. If he got were aware of its implications for the CRL business case, he could arrange for the report to be rewritten to suit his agenda.
Its already published but the NZTA were fairly clear about just because it was done for them that it doesn’t mean they agree with it.
Yeah thanks for the research Matt. But freakin hell, can we have this regime investigated on 60 minutes or Inside New Zealand. It really is sickening that costs or benefits aren’t really any concern to this agency, while pretty much every other govt dept. lives and dies by these values.
This is why I think the problem with transport policy in NZ is not just with the politics, but the profession itself. Why were the analyses so wrong? Huge optimism bias?
Money quotes:
“Travel-time savings on Greenlane West approaching the interchange do not appear to have been achieved.” Are they ever achieved in urban environments with latent demand? Economics says no!
“Predicted traffic growth has not been realised …” Hmm … maybe NZTA should explore that idea a little more!
Do you think induced demand theories hold true when you have flat-lined traffic growth? Now there’s an interesting can of worms to explore….
It is still clear that what we invest in will grow, we have seen that over the last decade with trains, we will really see it soon on the buses as well, and we are well used to it with roads. What is now different is will the road investment just help steady a decline or not? The rule of inducement doesn’t alter but what is the base level and what sort quantum of change might we see is the question.
The second half of last century saw road building unable to keep up with new car and journey volumes, and often had modellers fail to predict such high numbers. But that era has ended, as well as the fact that we have an existing resource of roads built during that period that is more than capable of handling the new steady state. Especially if we invest to meet the new demand people are showing for passenger transit.
So the question is; why don’t we embrace this opportunity and see how much we can transform movement patterns in Auckland by offering much improved alternatives to driving as a the best way to improve the experience for that majority still driving?
There’s an apparent dichotomy here in that “travel time savings on Greenlane West do not appear to have been achieved” ..yet “predicted traffic growth has not been realised”. Des this mean the issue is that the project failed to deliver the expected increase in capacity.. i.e. growth in peak demand rather than traffic generally?
Incidentally if I read it right, the auditors’ comments suggest the benefits include “unexpected accident benefits” of $ 12 m.. So the BCR was really only 0.5 ($ 17.4 m / $ 34.7 m)?