In an 1879 essay, Francis Walker tried to explain “why economists tend to be in bad odour amongst real people.” Walker, who went on to become the first president of the American Economic Association, argued that it was partly because economists disregard “…the customs and beliefs that tie individuals to their occupations and locations and lead them to act in ways contrary to the predictions of economic theory.”  Frank et al  1999

As some of you may be aware, Christchurch City Council has applied for NZTA funding to develop the network of major cycleways illustrated below.

majcycroutes-may14

This economic appraisal of this investment is discussed in detail in this blog post. The benefit-cost ratio for the $160 million investment was estimated using NZTA’s Economic Evaluation Manual to be 7:1, with the benefits of the project breaking down as follows.

chchcyclemodelbenefits

All well and good.

Until about a week ago when two economists from the University of Canterbury, namely Glenn Boyle and James Hill, released their review of the business case for the major cycleways network. Based on their review, Boyle and Hill conclude the actual BCR is more likely to range from 0.7 – 1.6. They reached this conclusion for the following reasons:

  1. Fuel prices – Rather than the 40% increase in real fuel prices assumed in the business case, Boyle and Hill suggest a more reasonable assumption is constant fuel prices. This reduces the BCR to 6.0.
  2. Time savings – Boyle and Hill calculate the average time saving per vehicle trip and conclude that because it less than 6 seconds, that these benefits should be discounted. This reduces the BCR to 4.6.
  3. Safety benefits – Boyle and Hill argue that the procedure used to calculate safety benefits is designed only for small projects costing less than $5 million. Removing these benefits reduces the BCR to 3.8
  4. Health and environmental benefits – Boyle and Hill argue that the procedure used to calculate health and environmental benefits is designed only for small projects costing less than $5 million. Re-calculating these benefits reduces the BCR to 0.98 – 2.3
  5. Discount rate – the business cases uses a 6% discount rate; Boyle and Hill argue “market realities suggest this is probably too low”. Using a discount rate of 8% reduces the benefits to 0.7-1.6.

Oh dear.

I’ve subsequently reviewed Boyle and Hill’s review. My general conclusion is that while they make some valid points, they miss the mark in the places that matter. This in turn means that their conclusions are at best unsubstantiated and at worst simply wrong.

The key issues I find with their analysis are summarised below.

The first issue relates to their grounds for dismissing time savings. First, Boyle and Hill the divide total (estimated) time savings by the total (forecast) number of vehicle trips in Christchurch so as to calculate the average time saving per vehicle trip. They then reason that because the average time saving per trip is ~6 seconds, then the time saving benefits are too trivial to be included in the business case.

This analysis smacks of the sort of erroneous logical reasoning that one would critique in a first year statistics course.

Consider the statistical distribution of time savings that might result from cycle investment. It’s reasonable to suggest these savings will not be distributed evenly.

More specifically, the time savings will accrue primarily to vehicle trips that occur within specific corridors and at specific times. That’s certainly what the modelling of cycle flows seems to indicate, as shown below. From this map one might expect very small time savings for vehicle trips in areas such as the airport, Brighton, and Banks Peninsula, with larger time savings for vehicle trips travelling in the peak direction on key radial corridors.

Christchurch cycle network demand map

The need to consider the distribution of time saving can be illustrated with a stylised example. Imagine a city where there are 1,000 vehicle trips, and where a proposed cycle investment will save 2 minutes for 50 vehicle trips, while the remaining 950 vehicles trips are unaffected. The average time saving in this city is only 6 seconds per vehicle trip, even if every vehicle trip affected by the investment actually saves 2 minutes.

The takeaway message is that the localised time savings in a large transport network cannot be accurately represented by the average time saving per vehicle trip. The latter metric may indeed obscure what are tangible savings for a small number of vehicle trips.

Or, to put it another way, if Boyle and Hill wanted an even more sensational figure, then they could have averaged time savings over all vehicle trips in New Zealand rather than just Christchurch, and concluded that the project would save less than half a second per trip. But that would be even more ridiculous.

Reductio ad absurdum; QED Boyle and Hill’s dismissal of time savings is unsubstantiated (NB: One could of course analyse the distribution of time savings and consider only those savings that were above a certain minimum threshold, but Boyle and Hill have not done this).

The second issue relates to their choice of an alternative discount rate. The major cycleways project is a transport investment which is seeking funding from NZTA’s land transport programme.

Let’s make this very clear: NZTA specify that a discount rate of 6% is to be used when undertaking economic appraisals of transport investments. It is therefore entirely appropriate for the business case to use a 6% discount rate. If Boyle and Hill have an issue with the discount rate that has been chosen by the NZTA, then they should raise those arguments in an appropriate forum – not pretend it’s an issue associated with the major cycleways project.

At this point it’s worth pausing for a moment and simply noting that if we add back in even 50% of the travel-time savings and use the 6% discount rate stipulated by NZTA, then the BCR for the cycleways project is likely to return to respectability – even if we accept all of their other points. And I don’t …

The third issue relates to their discussion of travel behaviour change. Boyle and Hill question the magnitude of travel behaviour change, but ignore that the business case takes a conservative view of behaviour change when compared with the results of stated preference surveys.

Stated preference surveys in Christchurch suggest that up to 30% of people would be willing to switch to cycling if sufficient safe infrastructure was provided. At present, the Census suggests that around 7% of people in Christchurch cycle to work, while the Household Travel Survey suggests that around 3% of total trips in the Canterbury region are made by bike.

The modelling did not conclude that cycle mode share would increase to 30%. It took a much more conservative view, which is that the cycleways would boost the number of cycling trips by 15-35%, which would imply an increase in mode share of 1-3% to between 5-10%.

A bit of international context would help here. Other cities that have invested in transformative changes to cycling infrastructure have experienced much larger increases in cycling mode share. For example, the Dutch only started investing in safe, quality cycling infrastructure in the 1970s. Today, many Dutch (and Danish and Swedish) cities have cycle mode shares in the range of 20% to 40%. Moreover, more modern cities like Portland have achieved cycling mode shares approaching 10%. QED the travel behaviour change assumptions in the modelling are within the range of what we observe elsewhere.

The fourth issue is that Boyle and Hill have misread the EEM guidance on analysing cycling benefits. They claim that cycling benefits have been estimated using an inappropriate (“simplified”) procedure. However, that’s simply not true. The values used to calculate per-km benefits for cycling are part of the core EEM (Appendix A20, if anyone’s interested). QED it was appropriate for the original business case to include these benefits.

The fifth issue is that Boyle and Hill make non-standard assumptions about fuel prices. They criticise the modelling for assuming that real fuel prices will increase. However, the assumptions in the business case are based on modelling published by MBIE – who are hardly a bunch of peak-oil alarmists. Boyle and Hill’s critique basically boils down to “oil prices are around their historical norm”, therefore we should not assume any future increase in price.

As economists, they should know that past performance may not be a good guide to the future.

Boyle and Hill argue that crude oil futures quotes are expecting current prices will persist, although I understand these contracts 1) typically extend out only for the next decade or so, and 2) the liquidity is fairly low in more distant future years. In contrast, NZTA stipulates a 30 year evaluation period for similar transport projects. Moreover, unless Boyle and Hill have an alternative (forward-looking) model of fuel prices, as well as evidence that their model is more accurate than MBIE’s, then their objection to the fuel price assumptions used in the business case is somewhat vacuous.

If, as it seems, Boyle and Hill’s real target is NZTA’s evaluation methods, then their critique of the cycleways is at the very least misplaced. Christchurch is proposing to invest a decent amount in cycleways, but that expenditure is dwarfed by state highway spending. If, for example, Boyle and Hill applied the same attention to the $11 billion RoNS’ programme, then they’d find some projects which start out with BCRs less than 1.0, even with more uncertain agglomeration / wider economic impact benefits included from the outset.

MoT state highway BCRs 2005-2012

A serious investigation would have at least considered this wider transport investment context, before honing in on the cycleways project as perhaps a case study. And even then it’s a relatively non-representative (and unimportant) choice of case study.

So where does this leave us?

Well, if the aim of Boyle and Hill was to create clickbait for anti-cycling neanderthals, then they can rest happy in the knowledge that they have done their job exceptionally well.

However, if they wanted to foster more informed public debate on the merits of the major cycleways project, or the business cases for transport investment in general, then they have clearly and demonstrably failed. It’s especially unfortunate their review comes across as a deliberately sensationalist hatchet job with largely unsubstantiated and/or incorrect conclusions. By extension, their review does not – as they claim – call into doubt the key finding of the original business case, i.e. the investment in major cycleways represents a relatively effective transport investment. For these reason I wouldn’t expect their review to hold much sway with Christchurch City Council and/or the NZTA.

Postscript: While Boyle and Hill’s review is, in my opinion, “in bad odour”, it is encouraging that the business cases for transport investment are receiving more attention from the wider economics profession in New Zealand. I’d certainly encourage Boyle and Hill to pursue this new found interest further, and would welcome them scrutinising the business cases for RoNS projects, many of which cost in the billions of dollars and start with BCRs less than 2.0. That’s really where the real money is being spent, and it’s where the economic evidence is the weakest.

Share this

54 comments

  1. I can’t believe the amount of anti-cycling bullshite that some people spring up. You don’t even have to be a economist to know that a cycling culture will be far superior in Christchurch. I have spent 1hour 40 minutes commuting on two buses from my college in the CBD to my flat in the West of the city, despite being only 5km. While my friend cycled the same distance in just 35 minutes.
    My class tutor even excitedly told us about how her vegetable produce company, who gets their food from local growers, will now use cyclists to deliver their products.

    Just to add, I will hopefully be able to provide you blog readers a post regarding changes in Christchurch over the next weekend.

    1. yes I also find the antipathy that some people demonstrate towards cyclists somewhat curious. I don’t know what Boyle and Hill think about cycling, but their review strikes me as antagonistic, almost passive aggressive.

      Speaking for myself, I just enjoy the accessibility provided by all transport modes. I love driving, and applied my drivers license at the earliest possible age (17). I also love public transport in all of its various forms (except buses stuck in stop-start traffic – that’s almost worse than death). I love cycling and walking as well.

      It just strikes me as odd when people view the world from a mutually exclusive “modal” perspective. Cyclists are drivers too, and vice versa.

  2. Great review, Stu!

    One thing that occurred to me while reading it: In several of their critiques, Boyle and Hill are basically accusing the cycle modellers of professional malpractice. Using inappropriate assumptions from the EEM and so on and so forth.

    As you note, that seems to be baseless as it is based on Boyle and Hill’s misreading of the relevant sections of the EEM. It’s really not good practice to accuse someone else of malpractice when you can’t even be bothered to RTFM properly.

  3. I hope Hill and Boyle feel truly slapped down reading this, as it seems they deserve it.

    I’d be interested to understand who funded this twaddle. Are we talking public, or do we have privately funded (or directed) interests at work…?

    Unfortunately neither will be a good answer given the quality of the work.

    1. In their defense, I suspect it was funded from their own time/energy rather than externally, so their on OK ground then. At least I don’t remember seeing acknowledgments of external funding in their report.

      Indirectly, you might say that as their employer the University of Canterbury is the funder, but that’s a long bow to draw as I know there are transportation researchers at the UoC who would distance themselves from this sort of dirty academic publication.

  4. So if we take the 2 minute average timesaving on a drive to whangerei that the holiday highway provides, and then divide that by the total number of vehicle trips in new zealand, does that mean we can not build it now?

    1. Agreed – always nice to give people a right of reply, and based on my reading of their critique, it seems highly unlikely they’ve ever stumbled across this blog before. Want to give them a bell, Stu?

  5. It always amazes me how any half baked and incorrect analysis of either PT or cycling projects gets massive media attention yet this same media outlets will ignore any criticism of the RoNS

  6. It takes me an hour to get the bus to work in Auckland 10k from Auckland CBD (and costs $8 a day). It takes me an hour to drive into work unless I arrive in the CBD by 6.45am (and then cost me $17 to park for the day) (people also seem to forget that it takes time to find a parking space then walk from said parking space into the office). It takes me on average 20minutes to ride the 10k into town and lock my bike up pretty much at my desk (it is also free to ride, after the price of the bike of course).That’s and saving of 80minutes a day riding. And between $8 and $17(not taking into account fuel or depreciation) money saving a day. I also tend to stay at work later due to not having to get the bus or fight traffic. They win too.

    So, for me, saving nearly 7 hours and up to $85 a week seems like good economics.

    1. When teaching I ride to Uni because it is the cheapest (cost: some belly fat), quickest, and most convenient (point to point, at time of my choosing). Only counterfactuals are the risks I face because of unsafe infra, and sometimes the weather. The later in practice is not too much of a problem and the former increasingly looks like it will be getting sorted over the coming years.

      One clear benefit to the city of me driving is taking a car off that route and out of a central city parking space: leaving more capacity for others. Money spent making this a viable choice for more people clearly passes the sniff test for a positive cost benefit to society, even before the health and environmental externalities are added.

  7. Thanks Stu. It is articles like this that make Transportblog the ‘go to’ website regarding transport and urban design issues.

    1. thanks very much Brendon, I’m really glad you enjoyed the post and appreciate the positive feedback.

  8. Claiming it had a BCR of 7 just invited this. When I first started doing these in the late 80’s and early 90’s I worked with a guy who claimed there were only finite benefits in an area and if you claimed them all in one project the auditors would get you. I used to laugh at his idea but maybe there is some truth here. If they had got a result of 3.6 maybe nobody would have blinked.

    1. So one group overestimated and another underestimated, possibly to suit their own agendas?

      Surely not!

    2. Agreed to a certain extent. The problem really is that there is a great deal of uncertainty with BCRs generally, which are typically never reported, and further that uncertainty is greater in projects that have the potential to be transformative (i.e. change more peoples behaviour than you might expect). We’ve seen this with the Northern Busway and Britomart for example, where projections of ridership/use were greatly exceeded.

      The other problem is BCRs are perhaps better represented on a log scale, as the variability and uncertainty typically increases in proportion to the costs + benefits. If done, a value of 8 doesn’t look unlikely – indeed, the uncertainty is likely only symmetric on the log scale, not the linear scale. Further, this would at least give “negative BCRs” some meaning :p

      Reckon it’s good that the two economists took a crack at it – if only more would do so, we may get more light shed on some of the larger projects which are far worse than this one!

      1. You’re right that there’s a lot of uncertainty in BCR’s for transport projects in general, and there’s a lot of difficult-to-quantify benefits and costs that are left out. That’s why the most important thing is that they are done consistently from project to project, so that even if the end figure is no indication of whether to do a project or not, it at least lets us compare similar projects and prioritise the most worthwhile ones.

        But that’s why this review wasn’t great. Even if some of the ideas behind their suggestions had merit, most of them would result in the BCR being inconsistent and thus incomparable to BCR’s of other transport projects (eg. their suggestions of using a different discount rate, different fuel costs, different definition of time savings).

    3. So a BCR of 7 can never be true? Maybe so for a costly roading project, but that’s not always the case for other low-cost projects. Many low-cost road safety projects for example have often demonstrated huge BCRs, because one simple treatment (e.g. install a sign/markings) can tackle a blackspot with a very high social cost of crashes. Likewise, many cycling projects are relatively low-cost for the benefits they generate (esp. safety and health benefits). As it was, the Chch Cycleway analysis was already conservative on many fronts in its evaluation.

  9. Boyle and Hill also included 97 million worth of health disbenefits from the increased risk exposure of new cyclists.

    Given that they claimed only 9000 people would use the cycleway to get their gimmicky headline grabbing “you could buy everyone a car at that price” and about 7000 of those are claimed to be existing cyclists – this means that an extra 2000 cyclists are creating 97 million worth of disbenefits. That equals $48 500 of health disbenefits per cyclist.

    This ignores studies that show that the increased exposure to danger of cycling is outweighed by the health benefits.

    1. When I worked at Auckland City Council in the late 80’s I found a copy of a leaflet in a drawer someone had done against Robbie’s rail scheme that made the same claim. The guy had argued you could buy them all a car and build them a carpark for less than the cost of the rapid rail scheme. I am still waiting for my car.

      1. Next time a major road is proposed I’m going to object with “but you could buy everyone a car for that much”. That’s how stupid that argument is.

      1. Give me that car! I will sell it, and buy myself a new bike, pay off some of my mortgage, and go out for dinner twice every week for the next year.

        1. I guess that is actually the point. Sometimes we would rather they didn’t spend our money and just gave it back to us. (If they weren’t so religious I might move to Texas)

        2. Actually, I am very happy for them to spend (a good chunk of) my money on social housing, hospitals, police, fire men, road maintenance, cycleways, and all the other stuff that goes with good government. I am NOT heavily taxed and actually would be happier to pay more tax (w’ve got pretty low rates here in NZ) if it went to good things. Just not to another motorway please, we alredy done those to death.

  10. I might have missed it but were in the business model does it mention savings in fuel; surly that’s important when assessing a project; we use most of our oil on transport so any savings are significant in cutting our imports of oil. That should put the bike at the top of the tree when it comes to saving overseas funds.

    1. This would definitely be desirable! However, this does not happen much in practice in part due to the fact that transport models typically aren’t set up to calculate confidence intervals. The more complex they are, the longer it takes them to iterate to a solution… doing Monte Carlo analysis with a model that takes half a day to run sounds bloody awful.

      However, it is pretty common to “sensitivity test” the results for upper and lower bound assumptions. If you look at the original cycleway modelling (summarised here: http://cyclingchristchurch.co.nz/2015/02/22/show-me-the-money-the-economics-of-cycleways/) you will see that it did report sensitivity tests around different levels of behaviour change, fuel prices, etc.

      1. More specifically, take whatever the NZTA threshold is for a “high” score on economic efficiency and +-10 ;).

        1. Yes you’ve got it for the variance. The mean value is take whatever the NZTA threshold is for a high score then add 0.471. The score needs to meet the threshold and look suitably accurate.

  11. We could also buy everyone a bicycle, not build any more RoNs, and save ourselves 1,333,000 Suzuki Altos.

  12. My first reaction was the the discount rate looked a bit high. Of course, if we or they are projecting over a 30 year period, 6% is probably a safe assumption. But not 8%.

    They did not take into account maintenance costs on all those cars. I’ll forget about depreciation, wear and tear on roads, cost of police patrols, accident response, speed of emergency vehicle (St John’s) response in conditions of reduced congestion, ACC, and all the other things associated with highways that cost a lot of money. It will still come out with a higher BCR than they calculate.

    Good helmet-wearing behaviour on the part of cyclists helps the BCR a lot! 🙂 One head trauma can ruin your whole day.

    1. Helmets are overrated, at least for the average joe cycling to work. The Dutch for instance are doing perfectly fine without them.

      Lights on the other hand… Daylight savings ended, and most cyclists I encounter are still riding without any lights or reflectors after sunset. They should make having lights on your bike compulsory, rather than wearing a helmet.

      1. Interesting in that the Queensland Parliamentary enquiry into cycling issues 28th recommendation was for front and rear lights. I’ve been looking for the evidence to support their claim that it reduces collisions but can’t find it. Helmets and hi-vis have never been shown to do very much (the British police have even gone so far as to call hi-vis camouflage in sun strike cases). http://www.parliament.qld.gov.au/documents/committees/THLGC/2013/INQ-CYC/rp-39-29Nov13.pdf

        1. From memory, the NZ Cycle Safety inquiry also recommended new lighting standards for bikes so from what I hear we may get some new regulations on that sometime soon here as well.

      2. Very interesting. In my opinion, I believe a little protection is better than none. Assuming the price of two similar cars are the same, but one has airbags, but the other one doesn’t, which would you choose? I would never ever consider riding without a helmet, but that *might* be because I was born after the helmet law was passed.

        As for front and rear lights, that is a no brainer. If you don’t have front and rear lights on at night, you are simply looking for trouble, especially if you wear black.

  13. The critique is correct. Traffic systems, just as in computer systems, these are queuing problems and the average delay does not matter so much as the peak ( worst) times, to users.

  14. By average, do they mean “mean” time savings, or median time savings. However, either way, it is still a bit farcical. If it is median, it could be that some vehicle trips *could* get longer, and some get shorter, and some doesn’t change, however, the “median” driver is very, very marginally better off (not really). If the average is the “mean”, then extreme values could affect the results, therefore, a 5 minute (hypothetical) time saving is going to outweigh a two minute increase in journey times.

    In my opinion, the time savings are likely to be very, very right skewed, however I don’t have any information on it. This is because I believe congestion won’t decrease as bike lanes would just encourage more trips, and drivers might dwell longer when turning left/right in order to give way to cyclists.

  15. I am surprised that no-one (or have I missed it?) has factored in possible savings in roading infrastructure and maintenance if there is an increase in bike use. Surely this must boost the case.

    1. Glen,

      Thanks for your reply. I’ve read your review of our review (there’s a Russian doll situation emerging here). Unfortunately your response doesn’t address the issues outlined in my post above. Let me try and explain why:

      1. Distribution of time savings – It is standard practice to count all time savings, regardless of length. Threshold analysis of time savings (i.e. not counting small savings) is non-standard practice, and would need to be reflected by a change in NZTA’s EEM. I note that as paid researchers undertaking a formal/public review, you could have requested the modelling data underpinning the analysis (e.g. via an OIA) and re-estimated the time savings accordingly. Finally, I don’t think it’s TB’s role (as unpaid observers) to explain what should be *obvious*, i.e. the distribution of time savings will be heavily skewed. Your analysis of this issue was sloppy, IMO.
      2. Discount rate – It is standard practice to use a 6% discount rate with 4% and 8% as sensitivity tests. Now I appreciate that you think a 6% discount rate is too low. NZTA have, however, undertaken considerable research into this issue and concluded otherwise (e.g. http://www.nzta.govt.nz/resources/research/reports/392/docs/392.pdf). Ergo the 6% discount rate should be used.
      3. Travel behaviour change – I think you may be confusing mode share with mode shift? Stated preference surveys show that up to 27-30% of the total population would cycle in the presence of good facilities. The business case assumes only 2.5% (i.e. one tenth) of current journeys actually do shift in response to the investment. This is much lower than the 30% indicated in the SP surveys.
      4. Mis-reading of EEM procedures – Appendix A20 of the EEM is the full procedures. This appendix contains no reference to the rate of benefits varying by project size. I think you are making an elementary logic error: Effectively, there are some methodological overlaps between the full procedures (A20) and the simplified procedures (SP), but this does not mean the caveats that apply to the SP also apply to A20. Moreover the caveats which do apply to the SP are associated with demand forecasting, rather than benefits per kilometre cycled. There is, moreover, no methodological issue with the BC blending the simplified and full procedures, where appropriate.
      5. Fuel prices – I’d encourage you to engage with MBIE on this issue. I tend to disagree with your suggestion that a 40% increase from current prices over the next 40 years is unlikely.

      Thanks again for engaging with this post – although one can’t help but feel that much of these issues would have better been dealt with before your review was published.

    2. It’s worth looking for analogies when thinking about travel time savings – which are, after all, just increases in consumer surplus on the margin.

      Let’s consider, for example, a tariff policy that results in a very small increase in the cost of apples in the supermarket. Ten cents a kilogram, say.

      The benefits to local producers would be large, of course. So by your logic, which is to count large numbers but not small numbers, we should obviously include those in a CBA.

      But what about costs to consumers? Based on some rough calculations, the average Kiwi eats around 10-20 kilos of apples annually. This means that they’d have to pay an extra $1-2 a year. This seems quite inconsequential relative to the overall household budget – so perhaps, by your logic, we shouldn’t count it?

      Now, I highly doubt that you would advocate for protectionism on the grounds that the costs are too small for people to notice. So why are you so insistent that we must approach transport analysis that way?

  16. About 30 minutes ago I was nearly killed when an Anchor Milk truck decided to share my very narrow lane on a bridge which had a double yellow line and oncoming traffic (at 100 kph). If there had been adequate provision for that Anchor driver to overtake safely without him actually having to make a legal overtaking move, I wouldn’t be reporting him to the Police now.

    I’m afraid the intricacies of the economics of the cycle lanes in Christchurch aren’t of any great significance to me… and they probably wouldn’t be to any economist if the choice was between building cycle lanes and having a truck driven at him at 100 kph.

Leave a Reply

Your email address will not be published. Required fields are marked *