Last week, Auckland Transport sent out an email to HOP users stating that the terms of use for HOP had changed.

The link to ATs website show’d the terms of use but no information about what the changes actually were and I wasn’t the only one questioning what they were. So, I asked AT what changes they had made. This was their response.

We have changed multiple terms relating to:

  • the reasons for which we will cancel a card,
  • the length of time before a card “expires” changing from two years to six
  • terms relating to the Areas of Liability to be more easily understood by customers
  • the overall wording of the terms have been made clearer
  • formalised the timeframes for notifying the public of fare changes

The intent was to make the terms more customer friendly and simpler.

AT have now updated the website with these details (although it’s right at the bottom of the page). Hopefully they think to add this from the start next time.

While we’re on the topic of HOP, there have been a few articles in the last few days about the issue of online top up money expiring if you haven’t tagged on within 60 days of the transaction. The articles make out that AT are effectively stealing money from customer. Here’s what AT have to say about it.

“When you complete a transaction online, an action is added to the system which requires the HOP card to be tagged on before the action can be completed, which is when the balance is officially loaded on to the card,” AT’s James Ireland said. “Several times a day the new actions get added to the list, which is stored in every tagging machine.

“The main reason for the 60 day tag on limit is to ensure the amount of data held does not cause the tagging system to slow down, as it has to search the database when each card is tagged on.

“Between May 2016 and April 2017 $342,000 worth of transactions expired due to not being used within the 60 day period,” Ireland said.

That didn’t reflect the amount kept by AT, as it also included money “reinstated” when people complained.

“If someone does not tag on within 60 days, they can call us and we will reinstate the money.”

This is pretty inconvenient for customers, especially those who are not regulars and therefore might be put off using PT. However, at the same time it seems like it’s one of those tricky issues that will have bad outcomes either way. As ATs suggest, and this post points out, without this process the tagging machines would need to store and sort through a lot more data and that would make journeys slower for a lot more people.

A typical HOP transaction takes around 350ms to occur – in this time the card is read, the database queried to see if the card is valid or blocked, the top up database is checked to see if a top up balance needs to be applied to the card, and lastly the new balance is written back to the card. Every step of this process takes time, and time is critical. If transaction times were doubled to 700ms for example it would cause considerable delays to the tag on process and would create significant delays for people boarding their bus.

As it is I think we need to be looking at ways to further improve boarding times on busy routes with features such as off board ticketing and tag on/off. It’s just imperative that AT make the process for recovering any funds easy. It would also help if made it clear when topping up online that you need to tag on. Further, since you have to have a registered account to top up online, perhaps AT could have some sort of communication process for those who are nearing their top up expiring to go and use the card.

Finally, to put the $342k affected into perspective, AT have over $17 million in HOP balances in their account to the end of July (the unsettled relates to the introduction of Simplified Fares last year.

Share this

51 comments

  1. Say each of these $340,000 users has $20 in credit (perhaps a reasonable guess if infrequent users are reluctant to add heaps to a card they don’t use) that’s 17,000 extra accounts out of, what?, a million?

    Hard to see how 1.7% additional accounts slows things down much particularly with growth in active use over time.

    If it does slow things down, things are going to get real slow with continued PT growth.

    There must be a technological option of increased the power of the processing. If 1.7% really causes a problem then increased active use will and current processing capability will have to be upgraded.

    Hard to see that processing times are the real issue here. Maybe it’s something about databases being unable to talk to each other except through use of the card?

    1. Guess it depends on how they are optimising access to the backend. If this means the 98% of users save 1+ sec of access time, then I can see why they made the optimisation.

      More processing is not always the answer. Network latency, the type of equipment they have invested in, etc.

      The cost to upgrade this may have been higher than the benefit. Of course bad PR has it’s own cost, so they might now push a fixed to a higher priority.

  2. Why is there a delay in adding the money to your card in the first place? Why not load it on as soon as your credit card it charged. Tagging on to load it just seems to focus the load and cause issues like this.

      1. Thanks for that link. Steve’s post was well written. Also echoed my own frustration at the misleading article. Six years (formerly two) before the credit expires…

    1. The balance is stored on the physical card. So until you present the physical card somewhere, an online topup can’t be applied.

  3. If AT don’t want to store payments for more than 60 days in their data file they could simply credit the money back to your credit card after 60 days. They didn’t do that, they kept the money instead. If you complain they reinstate it, if you don’t they keep it. That is the same as saying they if you don’t notice they steal it and if you do catch them out they give it back.

    1. AT like to keep your money. Like the $10 they charge you for a replacement card when they cancel your existing one for no good reason.

      1. Jimbo try getting a replacement card that is delaminating Newmarket would have given me new for free but they didn’t have any in stock and at Britomart had them and they wanted me to spend $10 for a card replacement that looks like it has had a bad sunburn

    2. I strongly recommend reading Steve Biddle’s post, linked by conan gorbey – Scroll to the end of the article for the example of click and collect.

      I’m thinking that the trust account holding the money must be interest bearing, so since they’re always prepared to give you your money, they can lower their opex (due to interest earned) and still be innocent of theft…

  4. Can’t they just add a “needs top up” flag column to the first cards datatable? That way, they can do the “card valid” and “card needs topup” checks all at once. Is there anyone with technical knowledge who knows the system who knows why it isn’t been done like this already?

    1. Even if they don’t do that – a simple lookup into a topups table should take about the same time to execute regardless of the tables size – assuming the table is indexed properly…
      I’m guessing the real issue is storage size restrictions on those devices, that would make more sense…

    2. Yes, that’s essentially right if the databases are built properly, and the software on the back end isn’t too rigidly inflexible.

      Heck, even if the software on the back end is almost totally rigidly inflexible, you could still make “recover expired top-up” a button you press on your online account to have that expired top-up re-added to the queue.

    3. Indeed. If the code that runs on the terminal takes a non-negligible time to look up an entry in the topups list, then someone is guilty of sloppy programming.

      Reading between the lines here, I’d guess that the method used is a simple linear search through the whole list — the slowest algorithm known to man. That is the only method that would add meaningfully to execution time when some additional old entries are carried in the list.

      The solution isn’t to go to extreme lengths to keep the list short, it’s to write fit-for-purpose code in the first place.

      If I can diagnose this problem from afar, surely the penny should have dropped for someone in AT management, too? Isn’t this kind of basic IT stuff part of their core competency?

  5. On a closely related topic, I can now follow up with more information about children’s AT HOP cards. At the time of writing the post about AT HOP for children and youth: https://www.greaterauckland.org.nz/2017/05/17/children-youth-faring-hop-system/ I hadn’t yet received corrections for some inaccuracies in an OIA response. Here they are now:

    At the time I was asking (March), 37% of actively used cards registered to children (16,768 of 44,856) had been charged adult fares at least once during the life of the card. AT started contacting people whose cards had reverted this year to charging adult cards. They were not contacting the people whose cards had reverted earlier.

    In communications to me, AT was reviewing its policies, but I have heard nothing further. Does anyone know if any changes have come about, such as “flagging” children’s cards online that are charging adult rates but still show as children’s cards?

  6. This 60 days is a real pain for out of towners. I visit Auckland about three times a year and have a HOP card to get around with. Generally I top it up just after a visit while I remember what the balance is. I have just done that but wont be back for at least 60 days. Of course I could try and keep a zero balance at the end of a visit and top up just before I come – or it might be easier to bring my car and add to Auckland congestion!

    1. Hi Paul, this is a confusing area – If you top up online, you’re _partly_ right – The top-up needs to be loaded onto the card within 60 days (tag on/off). If you top up at a retailer or kiosk, then the balance is loaded immediately, no need to activate the top-up.

      In either case, the active balance is retained for six years since the last use.

  7. If they are really obsessed with processing time – maybe they should put some effort into re-educating people too young to remember, get in at the front door, and out at the back (Old Fogey alert)

  8. I live in Wellington and bought a TOP card in January. I used it several times in January but have not used it or been back to Auckland since January. When I return to Auckland (probably in December this year) can I use it again with the money I still have on it? By the way I put my money on not online but at the AT desk at Britomart. Can someone help me here as this issue affects me personally. I also don’t think I am the only non-Aucklander with a HOP card (its a good deal for any non Aucklander who often visits and uses public transport in Auckland)

    1. This issue does not affect you personally.

      It only affects someone who tops up online, and then, after topping up, does not use their Hop card to tag on to a ride within the next 60 days.

      If you top up in person at a machine, this issue doesn’t affect you.

      If you top up online, and then within 60 days you tag on your Hop card on a bus, train or ferry, this issue doesn’t affect you.

    2. “the length of time before a card “expires” changing from two years to six”

      So long as you come back before 2023 you should be good!

  9. Fair enough, if such “top-up data” saved in every HOP reader slows down the system too much…

    Why not save ALL such “top-up data” that’s passed 60 days, on every “HOP Ticket & Top-up” machine? Spell it out to users along the line that “Online top-ups shall be uploaded to your card if you use your card within 60 days; beyond 60 days, top-ups shall only be added to your card upon presentation at any “Top-up machine”.

    In fact, is it possible for the “HOP Ticket & Top-up” machines to be upgraded as say, a “Service Point”? One that allows a user to check the “health and status” of his/her card… Kinda like these:
    https://www.octopus.com.hk/en/consumer/card-replacement/octopus-service-points/index.html

  10. I just had a look at my card online. Last used 15/08/2015.

    I use it spasmodically when the bus routes go near wherever I am working. Was ok when I was working down at the Marina, and my next location was easy walking distance there + bus home.

    But in 2014, I started working on the other side of Newton Gully, so there was no public transport worth mentioning that goes near it. You can spend 35 minutes on two buses and almost as much walking as just walking it. As I get older and more creaky (especially with an arthritic toe and a stent) I started taking the car.

    Moved to work down at the bottom end of Parnell. But since the Green Link changed its route in 2011/12 instead of taking 15 minutes from Grey Lynn to near the university, it now takes 40 minutes wending through Newmarket and Parnell to get to my current work.

    There might be years between my uses of the card now. And even then I usually use change if I can. The reason why was simply because I never know if the topups to my card ever work. The only one that ever did was when I drove down to Newmarket station and explicitly put $50 on. That was in 2013. The previous two attempts to do it online didn’t.

    I’d like to just pop a few hundred on the card, but there doesn’t appear to be an easy or convenient way to do that.

    1. Yes unless the card was faulty, I suspect you either tagged on too early before the tag machines had got updated with your topup (ready to apply to your card), or you took more than 60 days before using it after that online topup…OR the 2nd journey was more than ~60 days after that first “too early” one!

    2. Ummm, why don’t you take the Green Link bus anti clockwise via Victoria Park, rather than the long way round via Newmarket? Or am I missing something here?

    3. Why don’t you register your card online and then you can check what balance it has anytime (and top up online ahead of needing it too)? I live in Chch so don’t use my HOP card that often, but it’s pretty straightforward to check my balance ahead of a pending visit and add money if I need to.

  11. Steve Biddle says: “An analogy of this would be to compare it to ordering and paying for a product online from a click and collect retailer but never actually going to the store to pick it up.”

    I’d say: “An analogy of this would be for the only store in town to say that to buy any goods you can pay in cash – at very high mark-ups. Multiple times, in fact, if you’re wanting several items [no integrated fares]. Or, you can purchase a $10 card from the shop, which is non refundable. You can load money onto your account online, but you may have to wait up to 72 hours before you can buy anything. And if you take longer than 60 days to buy something, your money will no longer be in your account. Hopefully you will remember that you put money into the account – because if you do remember, you can ask for it to be put back into your account.”

    I have to keep 5 cards topped up (including spares for the children we “carpool by bus” places – which is against the rules!). Managing the 72-hour and 60 day rules makes my job very difficult. We can fly to the moon but we can’t even have a user friendly bus card. I can’t believe the number of people excusing this substandard system.

    1. Why not auto top up? The only problem then is if your credit card expires they will cancel all 5 cards without warning and you will only find out after you go to use them.

      1. For some trips I’m now sending the 11-year-old and a friend off on the bus by themselves. Auto-top-up is fine when you pop it back into your own wallet. It would be just another worry to think that if they leave a card on the bus and I forget to collect back the cards, that it could be costing my visa big time until I realise.

        And yes, the auto-top-up with an expired credit card leading to a blacklisted card put me off.

        1. Auto top up sends you an email when it has charged your card now – and apparently also advises when the card charge is unsuccessful. I definitely recommend giving it a go. I usually notice the physical card top up on the bus before I even receive the email.

    2. I don’t get the 72 hour rule yet alone the 60 days. Example. I am an infrequent user so when I found my balance was about 0 when getting off bus I topped up online. When I went to use my card later that day my top up wasn’t there yet so went negative I assume. I guess my top up will have expired now (60+) days. This cant be an unusual pattern for infrequent users. A year would make more sense. Or just pay it back.

    3. I was under the impression that refunds were indeed possible, per their T&C: https://at.govt.nz/about-us/about-this-site/terms-conditions/at-hop-card-terms-of-use/

      Also I’m forced to disagree with your analogy, which I think was unfair. A click and collect retailer always has a delay before you can collect your item, often 24 hours or more. In the case of PB Tech (for example), it could be two working days or more.

      I do agree that the situation is ridiculous. I don’t buy that the system would slow down by having the unactivated transactions uploaded to each tag machine. As one of the commentators from Steve’s post highlighted, 2 million cards = a flat uncompressed file of about 32MB. Right now I’m playing with a microcontroller development board (ARM Cortex M3 running @ 72MHz on a prototyping board for only $4 US) that could in theory parse that file in about 0.3 seconds. Surely they’re using a more capable processor than the low-end dev board I’m using.

  12. Yes the combination of system limitation and customer service is bad really. Now look at all the bad PR with those news articles. Should be able to refund the credit cards the amount almost automatically surely after the 60 days, then you are more likely to notice as well. I don’t think you get an email to say the funds have not been applied in time or you are about to run out of time?

    I’ve pretty much always used auto-topup with all the family’s cards so got it running sweet. Wouldn’t want to much around with manual online topups if can help it. Just need to watch the credit card or auto -topup expiry date (surely would get an email for this, haven’t had this situation yet). Sure wasn’t that easy to get my head around what was going on when the 2nd card I got somehow was faulty and never seemed to apply the auto-topups, was the only one like that.

    One thing to watch (unless they have changed it), if you check your auto topup settings for a card before getting it applied to a card (ie using the card at a tag on machine), it reverts/shows to the default settings of $1 etc and seems like (or actually isn’t) setup anymore ready to be applied.

    For visitors, casual & new users esp. from out of town that want to be organised a head of time etc all very hard.

    1. I got caught out with the auto top up expiring and not notifying me. Hopefully this has improved. I’ve now set the auto top up expiry to match my card expiry – so I only have 1 date to remember.

  13. So if I understand correctly the issue is we don’t want an unlimited amount of pending top-ups on the card readers on buses. That’s fair enough.

    But still no excuse for just silently dropping those top-ups. There must be better ways to handle that. It’s a limitation of the card readers, not the back-end. The pending top up can still be shown on things like the online transactions page. Maybe AT could allow users to reactivate the top-up before they return to town. Maybe AT can allow reactivating it by doing another small top-up. Or reimburse the top-up if it’s not used, (many hotels do with deposits).

    It sounds like a similar lame situation as with the failing auto-top-ups.

  14. The Thales RFID card readers used by AT uses an micro controller (ARM, similar to an early model iPhone IIRC). They can use FLASH RAM, and it could have 256 or 512 MB of it. Low level micros don’t have “databases” as such, just structures of data in RAM. A table with the last update date and amount for every card in the AT system would be less than 200 MB, and using a standard b-tree index would have a sub-millisecond lookup time. I suspect the real reasons is that Thales and AT don’t know what they are doing, and programmed a clumsy system. Other Thales implementation seemed to suffer similar problems. FLASH RAM has been cheap for many years, so there is no excuse for not including enough of it for every card in system

    1. Agree although a lot of the devices eg bus readers may be quite old so flash probably not quite so cheap then.
      They probably need some limit, but a year would seem more flexible. I cant imagine the number of people with a pending topup between 60 days and one year old is that long so why not keep those.

    2. As an embedded system design engineer, I’d expect they’re using something similar to an STM32F4 (2MB flash, 256KB RAM), coupled to SD or CF flash storage via SPI – Based upon a least cost approach, under $10 each in 1000 unit quantities. Would also allow them the freedom to write for bare metal or a realtime operating system (lots of RAM) in a variety of languages (though hopefully NOT Python. Or Java.).

      I pointed at the STM32F4, because STM Microelectronics guarantees that the part will be available for 10 years. Also, it’s cheap and powerful with great support.

  15. @Pete:

    The card doesn’t store all the data, its the processor and related systems on the reader which searches its database of pending transactions looking for your card number
    [i.e. the computer in the reader etc that are on the bus do the work not your HOP card].

    So its irrelevant how gutless the processor on the card is or isn’t – as it never does the work.

    @Sam Warburton & @Nicholas Lee:

    I don’t want to defend Thales or AT here for what is a pretty poor communication effort by them so far.
    [even if it is largely a slow news day media beat up at heart].

    But its a fundamental system design principle that you have to size for the expected peak number not the average number.

    [seems traffic planners do the same too hence all those massive roads with way more lanes than needed on average as they like to cater for the AM/PM peaks].

    This means you have to assume that with say 1 million HOP cards out there, a number of them will be using monthly passes, which requires more than the average number of tops ups as they need a top up once each month for the next months worth of passes to become active.

    That means, lets assume a moderately low 100,000 out of the 1 million cards on issue are using one monthly pass each [there are several monthly passes and you can buy more than one to have buses, trains a ferry passes].

    So they will generate 12 “top ups” a year, or 1.2 million top ups on their own [more than the number of cards on issue].

    Assume these top ups are randomly distributed across working days in the month [as these passes are mainly used during working days not weekends so a Monday-Friday tag on is more likely than a Saturday or Sunday tag with such a card with a monthly pass on it].
    So lets say thats 21 working days per month, then every working day there will be 1,200,000 / 261 [working] days a year = ~4600 top ups a working day are caused by those monthly passes to be processed.

    As we need to store 60 days worth of top-ups, [but thats only 42 working days in that 60 days] so thats 4600 * 42 = 193,000 top ups. Which are related to a single monthly pass that we would possibly need to store/handle, and that assumes we only have a single monthly pass on these 100,000 cards at any one time. Use two passes per card and you would double that number.

    Now I’ve excluded holidays, which will make the monthly pass list even larger as the top ups won’t be evenly spread over 261 days a year, more like 230 [which allows for public holidays and annual leave so reduces that by 31 days to 230]. Dividing 1.2 million over top ups over 230 realistic working days days gives a more likely [and higher] number of 5200 per day or 218,400 over 42 working days in that 60 day period.
    But lets go with the lower 193,000 figure for now.

    Add in the semi-regular and somewhat random at that level of top ups of the other 900,000 cards out there not using any monthly passes, so lets say, they’re topped up 5 times a year as an average given some cards are not used for 60 days after a top [or they wouldn’t even have this problem].

    So thats 900,000 * 5 / 365 (days per year) = ~12,300 per calendar day of regular top ups, times 60 days – that becomes 738,000 of regular top ups that could be pending at any one time on each and every reader in the system.

    Add the two numbers together of monthly pass top ups and regular top ups and you get nearly a million transactions in the pending list at any one time that have to be searched for each and every tag on as people are boarding the buses.

    While a search of a million records can be made fast [especially with good indexes on the data], 1 million top ups being searched for your cards number is not the tiny number that your 1.7% figure of 1 million cards suggests.

    The pending top ups list is probably quite similar in size in fact relative to the number of HOP cards on issue.

    If the size of the top up list will track similarly with the number of HOP cards on issue.
    Then the more HOP cards out there the bigger the 60 days pending transaction list will be.

    If numbers of cards grow [and especially regular use of monthly passes] or users do more frequent top ups than 5 per year [5 seems very low and its maybe more like a closer to 10 per year average or more for many of the regular active HOP card users].

    Then you have a list which is now getting closer to 2 million plus top-ups in the list.

    And remember this scan, check, confirm process all has to happen within 400 milliseconds or so at tag on time.

    So yeah, you can see why AT has limited the days a top up is left on the system to a arbitrary number like 60 days worth to ensure the system can cope with the essential transactions being able to get through.
    Leaving the laggards [those who don’t tag on with the 60 days to uplift their top up], behind.

    Maybe as the on bus readers get upgraded, that limit can be altered, but just as a chain is only a strong as its weakest link, you can only size your pending top ups list size wise so that it still works on the slowest reader on the [bus] network.

    And with thousands of buses in the fleet, upgrading them with newer readers over time will take time.

    Having said all that, AT certainly could manage it a lot better. Simple email reminders 7 days out might do the trick near when a top is due to expire is an obvious one.

    As is automatically putting the top up back on the list say the first 2 times after the 60 days are up.
    And with the email message going out to the registered card user email to remind folks to tag on to get their top up would eliminate probably 95% of the issues.

  16. Does the card need to expire? I’m 80 had my card couple of years. The card is still in good nick and I think that physically it should outlast me.

  17. And yet a city like London with probably around 6 million Oyster Card users can manage and the card stays active for years while still tagging on faster than Hop (and through things like wallets and handbags (most people don’t take their card out) and all of this was 7 years ago!

    1. yes oyster tags on much faster, but they severely limit the online topup process to minimise the size of the pending topups database. That wouldn’t work in Auckland where most people catch a bus.

  18. In London you can’t have an online topup applied on a bus at all! It has to be a physical station (tube, tram, dlr etc), and not just any station, when you topup you have to choose which station you will use next, and only readers at that station will know about your online top-up.

Leave a Reply