Massive outage exposes XT’s Achilles’ heel

[UPDATE: Some customers reported full or partial restoration of service from around 1pm. At 1.19pm, Telecom's official Twitter account weighed in with: "Some sites are back up, this is not yet stable. Please bear with us." As of 6.17pm, the telco said all of the 80,000 to 90,000 customers affected should have services restored.

Some cell sites became congested as traffic returned in a rush.

For those still experiencing problems, Telecom recommends turning off your cellphone, removing its battery, waiting several seconds, then reinserting it and restarting.]

Read also:
XT outage was beyond Third World - Tuanz
Telecom issues advisory for those still having trouble with XT
Was freebie data stick promo behind XT’s horror day?

Since 4.30am this morning, Telecom XT customers from Taupo to the southern tip of the country have been without voice, txt or mobile broadband service.

Details have been scant through Telecom's website or through official press statements,  but an email sent to corporate clients and obtained by NBR explains: "an unplanned restart of our Christchurch XT Mobile RNC switch which has resulted in the majority of XT cell sites south of Taupo not operating."

Restoration of service is expected "later today". 

One switch, two switch ...
Regardless of issues of 2G fallback (discussed below), one industry insider told NBR the problem was that Telecom only had two RNC switches ("a large bit of kit that coordinates all the traffic coming in and out of the network from cellsites"). 

Telecom has two RNCs - one for Taupo and north (located in Auckland), the other for everything south of Taupo (located in Christchurch). Each has multiple redundant parts, but apparently not the ability to cover for the other. Vodafone, by contrast, has five "which cover each other" and a sixth coming on stream during summer.

Telecom had no immediate comment on the RNC theory.

Achilles' heel
The catastrophic outage reveals the new network’s Achilles’ heel: XT's sole reliance on 3G spectrum.

Telecom has a second nationwide network, of course - it’s older CDMA service- but that’s incompatible with its shiny new XT.

Vodafone customers, by contrast, should be able to trip down to its older 2.5G/2G (GPRS/GSM) service when 3G stuffs out - for their phones are compatible with the telco’s older network.

I’ve consistently found XT faster in more places, as advertised, but this is one area where Vodafone, at least on paper, has the edge (I say at least on paper since a couple of brief Auckland outages have seen Vodafone’s 2G network simply overhelmed after 3G has gone down, nullifying any benefit of the backup).

2degrees also has a 3G/2G hybrid network (albeit with the 3G bit still under construction) in main centres, giving it the same fall-back benefit (and elsewhere, it piggybacks on Vodafone's infrastructure).

Above: blogger Lance Wiggs promotes a freestyle reinterpretation of Telecom's XT coverage map.

A so-so effort keeping customers in the loop
A number of people have complained about the lack of information on Telecom’s home page, or its XT home page (certainly, I couldn’t find any; I’m sure it’s there somewhere, but it should be front and centre).

A number of customers questioned Telecom's assertion that emergency calls could still be made. on XT mobiles that seemed "as dead as a brick" as one Twitter use put it.

Telecom’s Twitter team deserves more kudos. Updates have been constantly flowing through the social network (follow Telecom on Twitter here).

And customers of Telecom’s corporate division, Gen-i got sent detailed messages from chief executive Chris Quin at  9:41am, 11:04am, 12:16pm and 2:15 pm.

This article is tagged with the following keywords. Find out more about My Tags

Post Comment

25 Comments & Questions

Commenter icon key: Subscriber Verified

Uh hello? where was Vodafones 2G when their Auckland 3G network trashed itself for 24 hours? I seem to recall that it was as dead as a dodo too

My understanding is that Vodafones 2G network uses the same network core as their 3G network and as most network problems happen at the network core (as XT's is demonstrating today), their much vaunted 2G newtwork wouldnt be much use.

Great coverage but come on Chris check your facts and aim for balance

[Vodafone spokesman Paul Brislen says on this point: "3G steps down to GPRS down to GSM. It would be a rare event that would take out all three layers".

As noted in the article, sometimes this fallback gets overwhelmed.

Telecom has yet to respond to a request for comment on the cause of the outage - CK]

Reply
Share

This is great. We are all dying out here. This is the first time I experience this long outage. !!

Reply
Share

Spectrum is irrelevant.

If the cellsites are connected to the same switch or RNC (which they would for effective handover) and the switch or RNC goes down, everything goes down. There is no graceful failover.

As Mr Brislen points out, it is a rare event that takes out all the layers. Its called a switch failure. Seems like that is what happened today

[NBR's information is that Vodafone has five RNCs, and a degree of redundancy built in, whereas Telecom has two, one for Taupo south, one for Taupo north - CK]

Reply
Share

And it's back. In Wellington, at least.

Reply
Share

is why it has taken so long to recover the situation.

The network has been down from at least 6 am this morning.

My guess - an upgrade to the switch failed, insufficient testing carried out before the impact was noticed,with probably an untested DR process that has taken longer than expected to implement.

Having said that, the Telecom twitter team are showing the rest of the organisation how to handle comms in this kind of disaster event. Well done people!

Reply
Share

Miki, stop deluding yourself their twittering was pretty much awful. TelecomNZ tweeted that service would be restored at 10:30am, and then completely failed to deliver. That is the first rule in an outage crisis- don't make promises you can not keep.

Reply
Share

Well regardless of what Mr Brislen says when VFs network spat the dummy in Auckland (wasn't that for 24 hours??) 2G definitely didnt work for me or anyone I know.

It looks like Vodafone have wasted no time in contacting and "Educating" you - were they this proactive when their network died?

Reply
Share

I've been with Vodafone since 2001 and in that time I've only noticed the network down once and that was for no more than an hour. I reported it and Vodafone handled the issue very well.

Perhaps Telecom will build in a greater level of redundancy in future if Visionstream can find the staff to do it.

Reply
Share

The first rule in an outage crisis was to communicate with customers to let them know what is going on. With the Telecom Twitter team there is someone on the other end of the avatar.

Setting an expectation is fine - their error was not resetting that before 10am rolled around

Reply
Share

"Website maintenance

We're sorry, but we are temporarily unable to provide Broadband Usage Meter information. For further details and updates, please follow this link."

That's been the message since Friday for Broadband customers. Not being able to check usage is frustrating. Not being listed as a fault is glossing over the issue. And not answering calls to the help desk after 10 mins is doubly frustrating. Perhaps the helpdesk are on XT?

Reply
Share

My wife has just upgraded to XT. At the time she did, she was warned the battery life on her phone would be about 1.5 days (on standby). She and the other principals in her firm are charging their phones at least once a day.
Telecom's website says the phone in question has 2.4 hours talktime and 10 days standby battery life. When asked why it was discharging so quickly, they were told by the Telecom rep that it was because of XT (which I find hard to understand, but there's no other explanation).
As much as I love new technology, I'll be hanging on to my old CDMA phone for as long as possible - or switching to another network...

Reply
Share

Sorry to sound like Monty Python's Four Yorkshiremen but you're lucky. We do not get XT coverage here at all, despite Telecom claiming that we do and trying to get me to convert from the 0274 network through regular phone calls. When I ask them if they've got coverage here yet, they say 'we'll get back to you on that' and never do. It was only by going into a Telecom agent and actually pointing our house on their screen that I was able to confirm that they knew we had little or no coverage. I contacted their Contact Centre Manager about this and he said, 'Yes, I agree, we're selling a product that's actually not working at the moment.' And if you wonder why I don't just switch to Vodafone, their network isn't that good here either. The impact of all this is that I have to use an old phone and phone messages and texts don't always get through. Are we out in the middle of nowhere? Not exactly - I live and work 8 mins from NZ's largest shopping centre at Albany.

Reply
Share

Hi "disenfranchised" have you tried using a Vodafone phone that is compatible with BOTH their 3G frequencies? That is one that works on both 2100 & 900Mhz UMTS (WCDMA)? These phones are often advertised as working on the Extended 3G Network.

It might pay to give this a try - or if you have a friend that has one of these phones.

Here are a few examples of compatible Extended 3G phones:

* Vodem Stick
* Vodem Stick Pro
* Nokia 3120C
* Nokia 5320
* Nokia 5800
* Nokia 6121
* Nokia E63
* Nokia E71
* Nokia E75
* Nokia N85
* Nokia N96
* SE W715
* Sony Ericsson X1
* LG KS500

It might pay to also check the Vodafone Coverage maps:

http://www.vodafone.co.nz/coverage/

Reply
Share

I agree with anon R - plugging voda products does indeed give the impression that you're an employee not a fan boy. Either way its in pretty poor taste.

Reply
Share

Paddy, the reason is simple: most battery life quotes consider a mix of 2G and 3G use. Telecom ditched its GSM network after completing it but before switching it on (meh, what's $30m between friends) so there is no 2G in the Telecom network.

That means your phone is permanently on 3G.

That means it'll drain the battery so fast you could use it as a fan to keep you cool while you hunt for the next power outlet to charge it up.

iPhones are particularly prone to this. On Vodafone they'll last for a couple of days - on Telecom you'll be lucky to hit one full day of stand by.

Reply
Share

I was interrested in your comment about battery life. My crappy R7 normally goes for three days between charging. I text about 10- 20 per day, the odd bit of data amd perhaps 5 mins a day of voice calls. The fully charged battery did get a hammering today and I can only assume it was because the phone was transmitting at full power trying to find a working cell site! Ahh well, works now. All forgiven untill next time.

Reply
Share

I dont thinbk I've seen so many Vodafone staff trolling, making hay ever... I think you all know who you are and throwing stones when you live iin a glass house isnt terribly smart.

This comment stream seems to consist of vast amounts of disinformation interspersed with the odd helpful and constructive comment.

Reply
Share

It amazes me why the Telecom fanboys feel the need to assume that some posts on this topic are coming from Vodafone staff.

I for one do not work for Vodafone, however I am a happy Vodafone customers & have been for 8 years. I have posted what I consider to be helpful information, so I assume the last paragraph applies to me RE: helpful and constructive comment??

Obviously there are some questions to be answered about Telecom's XT Network, after a very major outage today (no matter who people may work for), following on from another outage only three weeks ago!!

People have a right to voice their opinion as you have done yourself.

Reply
Share

R I agree that the engineers at Telecom who sweated bullets fixing what looks like a pretty hairy and very complex outage, will be asking some tough questions, but I think the voda fanboys and voda staff should trad carefulkly - remember the march vodafone network meltdown in march?

Here's what the NBR wrote about it at the time, and I thnk any writers of comments about failover to 2G should re-read the article, here's an extract "The outage affected 3G voice calls and data. Voice calls should have automatically tripped down to 2.5G, but Mr Brislen says the 2.5G network was soon overwhelmed by the surge in traffic, with most calls going straight to voice mail (though two NBR writers found service dropped out altogether)."

So express away voda fanboys but just remember that the Voda network has faiuled in the past and it could do so again, just like XTs did today.

Reply
Share

First of all - at no time have I implied that Vodafone's Network is perfect or impervious to the odd technical issue - resulting in some kind of no service affecting it's customers.

Second - I am not plugging Vodafone products - I replied to a poster who was having coverage issues & I have given him some information that *may* help him out - of which I can't see anything wrong with! This is not in poor taste whatsoever, it's called "actually helping someone - unlike your useless post @vege.

Third - take off your Telecom hats & come to the realisation that there is no such thing as a perfect network & Telecom is no exception to this rule.

Reply
Share

R your post looks a lot like a plug, even if it is in the guise of attempting to help out someone. I dont support any of the telcos but I do feel that fanboys from all sides of the fence need to get a life

Reply
Share

I'm an XT customer but after today's event I can't help but hope the next time those 'creatives' at Saatchi's dream up a campaign like '...Faster In More Places', Telecom considers the pro's and cons especially when linked to technology that is so complex that it requires 24/7 performance monitoring. Mobile networks after all consist of lots and lots of hardware, software, transmission lines (microwave, fibre backhaul), and radio waves etc etc. Bearing these points in mind, when it comes to restoring services we would all be wise to cut Telecom some slack. But what was is becoming less palatable is the constant barrage of advertising messaging
on how great the network is when clearly it's certainly no better, if it's not available in two thirds of the country! I will now be following the performance of this network and comparing it to Telecom's advertising claims with interest.

Reply
Share

"until I make your enemies a footstool for your feet"...when people get hurt,specially the people under His grace are hurt. For the people that they hurt,for the workers got laid off,for the worries they made,for the people they rip off,for the tears that fell on each families cheeks, for the dreams they destroyed when the migrant workers were sent home,when they always rape us for their wickedness...for their are no rest for the wicked! For grace have been taken away...just pure bad karma guys,pure bad karma...many more to come

Reply
Share

Having worked around the world for various mobile networks, I find it rather pathetic looking at both sides of this Voda Vs Telecom argument hysterically waged by fans of either side.
Two rather Interesting points:

WCDMA is a redundant technology that very few Telco's operate in the world due to its limited growth potential in the technology sector. Spending the dollars Telecom did, was an utter waste. watch Vodafone's network speeds and coverage extend past telecoms in the coming years.

Telecom currently IS the FASTEST network in New Zealand in respect to mobile broadband ... thats about it, you can't argue apples with bananna's so all you Voda Fans stop arguing this with handset range, voda live blah blah blah.

after these two points I'm going to make one simple point. These telco giants in NZ respond to public demand (basic economics people) look at the mobile pricing in Australia as a prime example .. Cap plans, $10 Gb mobile broadband, free handsets worth more than $130 - $199. If the general NZ public knew any better and stopped wanking on about coverage, or how many minutes they receive. The market would become more competitive.

Point in point: Stop claiming to be king of the S$!7 heap ...

Reply
Share

As to the post about "Faster in more places" Maybe in reality they should have had the speedboat parked in the Auckland marina to really show speed. One user on a cellsite may be very fast indeed but when you have multiple users on it of course it slows down dramatically.
I have indeed found XT very fast in 'some' places but also use Vodafone and for overall reliability it hasn't let me down yet. By the way - as I found out over 2 months in Wanaka (as well as several other places), although there is very good XT coverage there it may be easier to use dialup as it'll be more reliable than an XT phone for calls & data!

So far in the South (and I do love it when Telecoms cell network goes down at a school I work at - such a laugh) there were several outages when Fibre got cut (over the years) and although Vodafone was working fine (Pre Telecom XT days too) the students seemed to feel as if their arms were chopped off with the inability to txt - yes some students were crying at lunchtime!

Reply
Share

Post New comment or question

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.

NZ Market Snapshot

Forex

Sym Price Change
USD 0.7743 -0.0004 -0.05%
AUD 0.9483 -0.0021 -0.22%
EUR 0.6328 -0.0008 -0.13%
GBP 0.4950 -0.0007 -0.14%
HKD 6.0052 -0.0021 -0.03%
JPY 92.5330 -0.0510 -0.06%

Commods

Commodity Price Change Time
Gold Index 1195.4 -2.890 2014-12-19T00:
Oil Brent 61.4 1.580 2014-12-19T00:
Oil Nymex 57.1 2.910 2014-12-19T00:
Silver Index 16.0 0.096 2014-12-19T00:

Indices

Symbol Open High Last %
NZX 50 5527.8 5553.7 5527.8 0.25%
NASDAQ 4752.6 4782.1 4748.4 0.36%
DAX 9901.3 9901.3 9811.1 -0.25%
DJI 17778.0 17874.0 17778.2 0.15%
FTSE 6466.0 6566.9 6466.0 1.23%
HKSE 23264.0 23478.9 23116.6 1.13%
NI225 17685.5 17692.6 17621.4 0.08%