Senior Telecom manager reveals extent of XT outage
[UPDATE, 7.15pm: NBR just spoke to Gen-i chief executive Chris Quin, who said Telecom chief executive Paul Reynolds will issue a major statement this evening. The statement is expected to answer customers' questions about compensation, and what steps are being taken to avoid a future outage. Mr Quin said the outage had been traced to Christchurch, and that the technical cause had been confirmed - but that it was not the same fault as caused the December 14 problems.]
Following the massive mobile outage that hit Telecom XT customers from Taupo to bluff from 10am this morning, the company's spin doctors said the problem would be solved by 1pm.
But five hours later, multiple XT customers were telling NBR they were still experiencing problems.
Telecom's spin doctors also said they couldn't say how many customers were affected, or the extent of the voice, text and data problems.
But an email from Chris Quin, chief executive of Telecom's Gen-i Australasia IT and telco services division, obtained by NBR, tells corporate customers a fuller story.
His wording implies that all XT users Taupo south suffered problems, though emails to NBR indicate that not all were effected.
"Bizarrely, my cellphone kept working whilst those around me failed – all (including mine) on the XT network," emailed Festivé Ltd's Gavin Holley, of Christchurch.
Mr Quin's email reads:
Today we experienced issues with our Christchurch XT Mobile RNC switch which has resulted in the degradation of the XT Mobile Network from Taupo south between approximately 10.30am and 12pm.
Users of the XT service from Taupo south will have experienced approximately one hour of degraded mobile service, and approximately 30 minutes of inability to use the XT service in the area.
Since 12pm we begun restoring the XT service, and full service will resume for all clients from Taupo southwards by 1.30pm.
XT services north of Taupo, and CDMA services nationwide, were not affected.
We are establishing the root cause of the degradation of the network.
We continue to understand that mobile is critical for your business, and we apologise and deeply regret any inconvenience or problems you are experiencing due to the interruption of the XT service.
We are aware that this is not the first time that we have experienced service degradation. Please accept my personal assurance that this issue was unexpected and was immediately escalated to the Telecom Executive team and our global partner.
I will communicate with you further with relevant information as it is available, and confirm that full restoration is continuing.
If you wish to discuss this further with me, or any of the Gen-i team, please let us know.
CEO Gen-i Australasia
One Gen-i corporate customer told NBR he gave Mr Quin credit for including his personal cellphone number at the end of his two messages (a third is expected later this evening). The Gen-i boss was also noteably more candid than his colleagues during XT's previous outage on December 14 - had included his email and cellphone number at the end of the message.
Telecom's Twitter account is advising those still suffering problems to remove their batteries, then reinsert them to reset their phones.
However, as of 2.59pm, at least a dozen customers tell NBR they still have no service after following this advice.
Telecom blames network congestion as cell sites come back online, and advises people to "keep trying".
Mr Quin has now sent a second message to customers. It reads:
This is the second email today to update you on the restoration of services for XT mobile clients operating between Taupo and the bottom of the South Island.
From approximately 10.30am this morning we have experienced issues with our Christchurch XT Mobile RNC switch which resulted in the degradation of the XT mobile network from Taupo south. As a result, many users in these regions have not been able to use XT services, including voice, SMS and mobile broadband.
Telecom has restored services for a significant amount of the XT mobile network that has been without service, however, due to the load on the network as services are restored and users commence using the network, we are seeing intermittent service for some clients.
XT services north of Taupo, and CDMA services nationwide, are unaffected.
For some XT clients still experiencing service issues, it may assist to try restarting their phone, by powering down then removing the battery for several seconds, then restarting the phone.
This issue has the highest level of executive escalation and our management and technical teams are working urgently to put a solution in place.
Once again, I apologise and deeply regret any inconvenience or problems you are experiencing due to the interruption of the XT service.
We are keeping our front line teams constantly updated, and will keep them informed on the resolution and next steps.
We will also keep you updated via email.
For further details, or to tell us how we can support your business immediately, please contact me or your Gen-i client manager.