Member log in

Telecom boss promises review after twin broadband blackouts - but for many, it ain't over yet

UPDATE 8.35am: Telecom has confirmed that tens of thousands are still suffering problems following Tuesday's twin blackouts, which hit hundreds of thousands.

"Although Telecom has restored service to the majority of its customers, a significant number of customers are receiving intermittent coverage and may find that their connection drops out.  It is estimated that the number of customers affected at any one time is in the tens of thousands," the company said in an 8.30am statement.

----------

UPDATE Dec 12, 7am:: Telecom Retail CEO Chris Quin is promising action after customers were hit by two major broadband blackouts yesterday.

 "We have a comprehensive review underway to ascertain how these issues can be minimised in the future and apologise unreservedly to customers for any disruption these outages caused in the interim," Mr Quin says.

At 1.15am this morning, Telecom issued a statement saying it had restored service to "the majority" of customers following an outage that began at 9.45pm Tuesday evening.

A number of Auckland and Hamilton customers have told NBR they still have no service this morning. 

Telecom says some might have to restart their modem to regain service. The readers say they tried that - waiting the regulation 30 seconds before switching back on - without any luck.

The outage was caused by a server problem.

The company says it was related to the one earlier in the day that saw two thirds of customers and about a quarter of eftpos machines lose service nationwide between 1am and 8.55am after planned maintenance at its giant Mayoral Drive exchange in the Auckland CBD went wrong. 

In NBR Comments (see below) and on Twitter, many customers were seething at the second loss of sevice within a day.

But one Auckland man took events in stride, tweeting:

And comedian Dai Henwood offered more humour from the trenches:


Details fleshed out
"The first outage related to a problem with a software upgrade on an Auckland-based server that is part of a system which handles just over 300,000 customers, representing approximately two-thirds of Telecom’s broadband customer base," Telecom said in a statement this morning.

"The problem meant customer modems were failing to authenticate when trying to communicate with the server. The Auckland server normally runs in parallel with a Christchurch based server and service was initially restored by removing the Auckland server from the system and running solely on the Christchurch server.

"Yesterday afternoon service resumed on both parallel servers, but last evening further problems were experienced with the Christchurch server overloading due to issues synchronising information between the two servers. To address this issue, the Christchurch server was taken off the system late last night, with internet traffic now all going through the Auckland server.  Telecom is urgently investigating why the synchronisation issues occurred."


UPDATE / Dec 11, 11.45pm: Telecom Retail CEO Chris Quin has just tweeted:

"Broadband services have just started restoring again, team staying in place through the night to monitor this closely...sorry again." 50,000 customers are back online. All should have service restored within an hour."


UPDATE  / Dec 11, 10.30pm: Telecom's broadband service has been hit by a fresh wave of problems.

Spokeswoman Joanne Jalfon told NBR ONLINE an outage hit at 9.45pm.

"The issue is believed to be related to an incident which affected broadband service early this morning," Ms Jalfon said.

The morning woes were blamed on a scheduled maintenance upgrade gone wrong at the company's largest exchange in central Auckland.

"Telecom's customer service team is responding to a high volume of calls and apologise to those who can not currently get through to speak to a staff member," Ms Jalfon said.

The support section of Telecom's website has been slow to acknowledge the problem.

But on social networks, on social media and in Comments below, numerous people around the country have complained their broadband has again blacked out.

Telecom's official Twitter account has been hit by a blizzard of complaints.

Via the social network, the company asked:

Many customers were exasperated their broadband had gone out for the second time in a day. 

"Internet down again tonight. Come on Telecom sort it out," and "So this is how Syria and Iran do it" were typical comments.


Telecom broadband back after seven-hour, nationwide outage

Dec 11, 8am: Telecom's broadband service was restored at 8am following a seven-hour outage. Affected customes should be back online within an hour.

The outage hit around two-thirds of the company's 500,000 or so broadband customers nationwide, spokeswoman Joanne Jalfon told NBR.

About a quarter of the country's eftpos machines were also taken offline.

A planned maintenance upgrade at the company's Mayoral Drive exchange in central Auckland caused the nationwide problem, Ms Jalfon said.

An exchange in Christchurch had taken load off the affected Mayoral Drive system.

Some customer might have to restart their modem to regain service.

A Telecom broadband outage on the morning of November 30 cut off around half the company's customers until 9.45am. That problem was also blamed on an upgrade gone wrong.

More by Chris Keall

Comments and questions
58

How would we feel if this were the telephone service?*
Now consider that for many internet is more important than telephony.

* the answer is that most of us have backup in the form of mobile phones. The real problem is where a failure is at a point which affects capacity on both networks.

We have mobile phones as backup but I don't pay the carrier to provide service that let's me Skype with family and friends and so my online gaming.

It's out AGAIN

Maybe Telecom should now change its advertising "Now's Good" to leave Telecom?

Maybe you should get your companies right... Nows Good is the marketing slogan for Telstraclear?

that's what anonymous is getting at. duh.

A Telecom rep came to my door last week trying to sell me a package. I asked them about their problems with email and outages and they stuttered through assuring me otherwise. Seven days later and they've since had two outages. Glad to say I sent the rep packing and stayed with Actrix. Never have problems!

Well, we like many, have migrated to VoIP. So, yes, it is very concerning it has the risk of losing voice services. You really wonder if technology is taking us backwards sometimes...

Telecom and others have always said that VOIP and other internet services are far less reliable as systems than the old POTS phone exchange. Not much could go wrong with the phone system. These new systems are incredibly complex and rely on very hi-tech hardware and software that is never standing still (always being upgraded) and sometimes fails. It would be nice if this kind of thing never happened, but don't bet your business on it. I heard some describe installing these kinds of upgrades as replacing the engine of a plane mid-flight...

I don't buy that. You design for resilience and fallback. You expect your service-provider to do the same. A failure is inevitable. A downtime measured in hours rather than minutes or seconds is not.

If your broadband connection is "mission critical", then get a back up. A router set up with 3G fail over is a no brainer in these days of pre pay mobile data packs. Also remember a UPS for when the power goes down at the house but stays up at the exchange.

3G internet would have been effected also because of the severity of the outage. 3G is only over the air until it gets to the nearest terra ferma backplane then its normal internet

3G internet is not affected ... outage was for broadband services.

Some customers have reported that service between 1am and 8am today was the most consistent they have ever experienced and that they were amazed that it only takes 7 hours to fix a fault.

One of the many hidden risks of cloud computing.

Our connection wasn't working normally again til mid afternoon so it was really closer to 12-14 hours of disruption!

How do u restart your modem? Is it the restart button at the back?

Switch off and unplug for about a minute you should be good after that

Anyone suffering again. Ours was up now it's down again. Mobile works but adsl is off again.....

It seems to be down again. Getting ***** off at Telecom. Have to use my Vodafone credit on my phone. Grrrr

Internet down again tonight. Come on Telecom sort it out.

yeah was having a good watch of some live program and suddenly the internet shuts! now using 3G

Mine's still not working!! So sick of Telecom. Will be changing companies ASAP.

Yep, haven't had these problems since I went to Vodafone. Looks like the Telebomb employees are being active with the dislike button ...

Our broadband is off again! What is going on!!!!

My Telecom ADSL is down in Christchurch again as well. It went down at about 10:00pm
It was also off until 10:30am this morning not 08:00am as reported

Telecom apologists go hide under your logs, it turns out they were teasing us, broadband's gone again (unless you have a smartphone or like me a 2 Degrees USB modem) Third world at best.

Broken here in Auckland, Takapuna. What a joke. Need Vodafone 3G so I can get online and up and going again! @TelecomNZ, fix your broken network. This is BS.

So this is how Syria and Iran do it

I for one admire and applaud Telecom's decision to hire the mentally challenged to carry out upgrades of their broadband nework. This is the kind of forward thinking that willl take New Zealand into the 21st century. Random disruptions to my internet service are a small price to pay in exchange for the smug feeling I get knowing that somewhere out there is a Telecom engineer wondering which end of the screwdriver to use.

Movies depicting the end of the world always start with lots of earthquakes. The modern equivalent is obviously lots of internet outages.

The first time I lost my broadband yesterday I thought perhaps I had been disconnected for persisting with Telecom over at least a credit on my officemail account that was down for 3 days last month.

Apparently senior management had decided their customers were not entitled to compensation in that instance as it was not their fault or something like that. Whats the excuse this time Telecom?

No internet in Queenstown yesterday morning and again this morning for two hours with Telecom. You need to credit customers accounts for this failure of service. Our 3g backup from Vodafone sucks the data that we have had to pay for to keep our business connected.

Back-up. Always have back-up. We happen to have Vodafone fixed broadband but also have Telecom USB modems, just in case.

Like many other places in New Zealand, internet access via Telecom is unavailable in Whitby -- for the second morning in a row.

Telecom: Please concentrate on being an excellent infrastructure company. Broadband "dial tone" is what we need, at the same levels of reliability as plain old telephone network. Most of us don't need nor want UFB, software upgrades or advanced functionality.

Telecom doth giveth (cheaper roaming) and doth taketh (outages) away.

Broken again in Takapuna on the Shore back to @VodafoneNZ 3G that is actually quite fast! Faster than my Telecom ADSL connection wish the data cap was bigger on it

I see Chorus at the top of the NBR Most Active Online Subscribers list but Telecom nowhere. Time to give the Telecom CEO a free subscription?

Here in Masterton, broadband has been sporadic for weeks. It goes down for just a few minutes, every day, and this has continued this morning. It's not quite bad enough to sack Telecom, but it's nonetheless a confounded nuisance.

Interesting - it goes down every day in Onehunga for about 5 minutes, perhaps two, or three times a day. Has done now for a couple of years.

On ya Telecom..........

Let's party like its the launch of XT!!!!!!!!! Useless!!!!!!!!!!!!!!!!!!!

Telecom charges like a wounded bull for consistently faulty services. Rebate please.

Any Compensation likely.....?

Hard to draw a "line in the sand" (Moutter's words) on broadband market share when Telecom fails to get the basics correct.

Time to support the challengers...Orcon, Callplus, etc.

Industrial sabotage by Vodafail to smokescreen Telecom's roaming data coup.

I've had enough of this BS from Telecom!

I'm off to Vodafone & I will advising all my friends & many Business contacts to do the same!

Not happy, not impressed & frustrated at these ongoing issues! How the hell can I do my Business, when my Internet is down??

BTW - Telecom Mobile coverage is non-existent where I live, so I can't use that as a backup!!!!!!!!!!!

Be careful who you jump ship to!

Vodafone had an outage of fixed line, broadband AND mobile back in August.

http://www.nbr.co.nz/article/nationwide-outage-hits-some-vodafone-3g-fixed-line-and-broadband-service-ck-125344

Orcon have a deal on today only where you can get $12 off there plan price for a year. Might be time to think about a change if you are with Telecom.

We have continued to have uninterrupted service from Actrix Networks. And even better, their customer service is fantastic in helping us sort out any of our user issues with mail, etc, in very non-technical speak. What a great illustration that biggest is not the best.

Actrix's marketing strategy:
- comment on NBR articles.
- that's it.

When I clicked the link to read the article this morning xtra cut out on me. Funny but not laughing.

Why does Telecom post on Twitter about internet outages.... FAILBLOG

Anyone got stats on the up/down time of Telecom and the other broadband providers? Are any of them more reliable than Telecom? Statisically please, not anecdotally. What about mobile providers? And how do they compare to overseas?

If this were a medical situation we'd be burying 500,000 or so people.

But it's not Des, is it Des? If it was a zebra, it would have stripes. Any connection. Um, nup. Make sense? Um, nup.

Service extends to telephones. Heavy rain kills our phone/fax/internet and a permanent solution seems unavailable.

To add insult to injury, Vector is going to lights, phone, monitored alarm, internet and fax on Thursday night. Great for those of us working across time zones. Not!

Sick of all you Aucklanders complaining. I have suffered Broadband outages right up to Thursday 13/12/12 lunchtime here in Ashburton.

Because we do not live in main centres we have to pay $10 p/m more for our telecom services.