Kiwirail’s recent decision to release the full business case for the third main has provided an interesting look into Auckland’s transport alternatives. As Matt outlined in his post, it’s strange that this project hasn’t been funded while the far costlier East-West Link is being fast-tracked:
The section of track between Westfield and Wiri is one of the most heavily used in the entire country. On top of serving both the Southern and Eastern lines, it is also sees more rail freight pass over it than any other stretch of track in NZ. It’s so busy that Kiwirail have declared “the current twin track configuration has reached the maximum operational capacity”“. That lack of capacity is resulting in delays to both passengers and freight. It’s long been known that the best solution to the issue issues between Westfield and Wiri is to build a third track to improve capacity. Yet bizarrely, the project still hasn’t been funded.
To be clear, there seems to be a good case for the third main – its costs are easily outweighed by benefits for freight users and public transport users. And it’s far better than the alternative of shifting more freight to road – that costs more than building the third main yet actually manages to worsen outcomes relative to doing nothing.
However, the business case also provided an interesting demonstration of how a poorly-designed multi-criteria analysis can misinform people about the impact of alternative options. Let’s take a closer look at that – but be warned, this is going to be a quite dry discussion!
This table summarises two different ways of measuring the impact of the project. The top lines show a multi-criteria analysis, or MCA, of the alternative options that the business case considered. As you can see, options are scored on a positive effect / negative effect scale relative to the Do-Minimum option, which is set to zero. Larger positive or negative effects on individual criteria are colour-coded, and then the impacts are added up to get an overall score and ranking.
Below this, the table reports the results of a cost benefit analysis, or CBA, of the options. The benefits of the project for freight users and public transport users have been estimated, and compared against upper bound and lower bound cost estimates. The resulting benefit-cost ratios, or BCRs, show the relative size of benefits versus costs. A BCR above one indicates that the project has more benefits than costs, while a BCR below one is a signal that you should consider doing something else with the money.
Here’s the problem: the MCA and the CBA don’t really line up with each other. The third main option has the highest BCR – in the range of 1.5 to 1.8 – but it’s only the second-ranked option on the MCA. The option to build both a third and a fourth main together has the second-highest BCR – in the range of 0.8 to 1.1 – but it’s the top-ranked option on the MCA. And the option with the third-highest BCR – no rail freight in the commuter peak – is ranked 8 out of 11 on the MCA.
As an economist who carries out these types of evaluation on a regular basis, this is a red flag. If the MCA is so wildly different than the CBA, it suggests that either:
- The CBA has missed some important things that it should have tried harder to measure; or
- The MCA framework hasn’t been designed very well.
In this case, it seems to be the latter. Let’s investigate why.
The first thing to note is that of the nine categories in the MCA:
- Three criteria relate directly to the benefits measured in the CBA – “Freight delays”, “PT Travel Times”, and “Reduced freight on road”
- Two criteria relate to benefits that could be measured as they represent flow-on impacts from the measured benefits – “Safety” (which is enhanced by getting trucks off the road) and “Economy” (which I presume relates to the productivity gains from faster freight)
- One criterion relates directly to the cost of the project – “Affordability”
- All options were scored as having no impact on one criterion, raising the question of why it was even included – “Cultural, social, and environmental effects”
- Two criteria are a bit fluffy but legitimately difficult to quantify – “Feasibility” and “Public / Stakeholders”.
In other words, the majority of the MCA categories are actually linked directly to the quantifiable costs and benefits that should be included in the BCR. This makes it even more puzzling that the MCA results were so wildly different than the CBA results.
We can start to see why this happened when we think about the way that costs and benefits are weighted in the MCA. Taken together, the three measurable categories of benefits included in the MCA get three times as much weight as the “affordability” criterion.
But we know from the CBA that this isn’t realistic! Measured benefits aren’t three times as large as costs for any of the options – in most cases, costs are a bit larger.
Basically, the MCA gets different results from the CBA because it places an arbitrarily high weighting on benefits, and an arbitrarily low weighting on costs. That’s why the fourth main option comes out ahead of the third main option: it has higher benefits, which are amplified by the MCA framework, and also higher costs, which are de-emphasised by the MCA.
Furthermore, a closer look at the “affordability” criterion reveals some serious problems with the way the options have been scored. As you can see, the Do-Minimum is set to zero, as it is the baseline against which other options are considered. The third and fourth main options are also set to zero, in spite of the fact that the cost estimates indicate that they will actually cost more. Meanwhile, the “longer freight trains” and “faster freight trains” options, which cost slightly less than the third main option, is scored as being mildly negative (“-“) for affordability.
It’s difficult to see how spending more can be equally affordable as spending less, but that’s what this analysis implies.
So what does all this mean?
As I said at the start, the third main is a vital project for relieving a bottleneck in Auckland’s transport system. The fourth main also seems like it could be a good idea – if benefits are higher than expected, or costs lower. It’s excellent that Kiwirail is starting to consider when that might be needed. But the results of the MCA don’t prove that we should prefer it over the third main!
More broadly, caution is needed when using MCA to assess projects – or when reading the results of said assessments and trying to figure out what we should do as a result. Used well, an MCA can be vital for assessing options in situations where a proper CBA isn’t practical. But it’s also easy to make mischief with MCAs.
For further reading: the Treasury’s Guide to Social Cost Benefit Analysis has a nice discussion of the strengths and weaknesses of MCA.