LIVE BLOG: Gen-i's XT outage update to clients

From around 12.30pm, I'll be live blogging Gen-i chief executive Chris Quin's webcast to around 200 clients of his Telecom services devision.

This wont be blow-by-blow.

Mr Quin has already sent a comprehensive series of emails on the subject so I won't repeat the basic facts (see his latest end of page), and we'll take the apologies as read.

I'll only add fresh facts about network restoration; details about the independent review (Who's doing it? Will it be made public?)

I'm paraphrasing Chris Quin's comments unless otherwise noted.

Telecom shares (NZX: TEL), which were at $2.44 mid-week, were down five cents to $2.36 in late afternoon trading.

1.28pm: Quin: Fault is not about RNCs - some UK telcos have four for whole country. But Telecom will install more RNCs, which is mainly part of the growth of the XT network as we see a high level of people moving across.

1.27pm: Question from NZ Post - should we keep our old CMDA phones? Quin: Contingency plans are important. It's a conversation that individual customers should have with Gen-i.

1.26pm: How does version 2 XT sim card work vs version 1 XT sim card? Quin: Most of what we've been doing around upgrading sim cards has been around international roaming, and making phones work better cellsites. Definitely an advantage to being on latest sim card. Plan to get all Gen-i clients there.

1.22pm: Question - why was Telecom network status page not updated until today? Quin - used Twitter, email, every method we could think of. Page may have been cleared overnight as updated.

1.21pm - Question  from Lance Wiggs - will independent inquiry look at cultural and structural issues as well as technical? Quin: Yes, operational as well as technical. Telecom has shown great culture. Staff rallied around. Absolutely we've got to look at process.

1.18pm: Will Telecom ask for compensation from partners and those who helped build network? Quin: No. Telecom will take accountability. Most important that has Telecom chosen best partners and is working closely with them.

1.15pm Question - Why no link between Auckland and Christchurch RNC? Quin - Some links, but putting 453 southern cellsites onto Auckland RNC could not be done; no network in world could do similar.

1.14pm: Question from Kiwibank - why not reserve capacity to bring all 5% of outstanding cellsites back online at same time? Quin: Not a capacity issue, more of a process issue. As gain confidence will bring more online at once.

1.13pm: Question from Juha Saarinen - handing out emergency CDMA phones? Quin: Yes, CDMA phones being put on planes or driven by staff to key clients.

1.13pm: Question: XT ever going to be an Apple-approved network? Quin - They don't approve networks as such but partners. It's well known that many use an iPhone on our network and indpendent reviews I've read say it works well. Currently no plan around delivery of iPhone. We've enjoyed supporting clients 

1.10pm: Question from Bill Bennett - Have XT's problems happened anywhere else in the world. Is the XT network unique? Quin: No, not unique problem. Chose Alcatel-Lucent because done similar networks before. But New Zealand is a unique country - long, thin, hilly, sparsely populated.

1.09pm: Question - will it be hours or days before service is restored? On the first day we thought it might be 1 or 1.30pm. [Now] we're going to be straight up and factual so we're currently not saying a restoration time. Even when every cellsite is back up we want to see sustained stability before stand-down crews.

1:08pm: Question from Chris Keall - Will results of independent review be made public? Quin: Yes we will make recommendations public.

1:07pm: Express Data - How long independent review take? Quin: Don't know yet. Wants time to include Gen-i customer feedback, but also quickly implement recommendations.

1.05pm: Question from Otago Chamber of Commerce - when service coming back in Dunedin. Quin: no time frame. Some sits back. CBD sites being prioritised.

1.03pm: Question - should I hold off upgrading rest of my staff to XT until Telecom resolves issues? Quin: We're going to restore service. We're being very open about technical steps being taken. Sit down with your client team and we'll explain why we're confident.

12.59pm: Porirua City Council - plans to build more redundancy? Quin: Question is more resiliency/capacity; how to cope with large mobile data loads. Yes, we are building more capacity - that's part of winning back your trust. 

12.57pm: Napier City Council - how can you say call completion in mid-90s when we're getting call dropouts? Quin: mid-90s is an average.

12.57pm: Question: How can be sure won't happen again. Quin: Any technology provider that says this will never happen again is naive. Have to make sure up-front about fix and preventing same thing occuring.

12:55pm: Question - still retire CDMA given outages? Quin: people are remembering CDMA's resiliency compared to XT's initial issues. Will still decommission CDMA in 2012. Issues will be addressed by then.

12.52pm: Question from Kiwibank: Why has Telecom not enabled in-country roaming so customers can use other company's networks during outage? Quin: Most phones not compatible between different networks.

12:.51pm: Question from Southland District Council - plans to add more RNCs [Radio Network Controllers]? Quin: All RNCs have redundant controllers. Telecom will add RNCs, and take other steps to increase resiliency. [Currently Telecom has two RNCs, Vodafone 6 - CK]

12.49pm: Question from Chris Keall - who will carry out the independent review. Quin: Telecom in process of engaging an independent commercial international player. Not signed contract yet. Will be a party independent of Telecom and its partners.

12.46pm Question from Farmside - how prioritise getting users back online? Quin: Combination of where mass residential areas are; where key Gen-i clients are; where events like concerts and race meetings are [Tell it to Queenstown! - CK]

12.45pm: Number of clients want to dual use CDMA phones. See your Gen-i client manager

12:44pm: Question - compensation? Quin: Focus on fixing problem, closely looking at. Will openly answer compensation issue "over the next little while".

12.42pm: Question: why XT problems Dec 14 and now Taupo south - why both times in that area? Quin: Network is in two regional areas. No link between the two outages; first case - corruption of software; current outage is hardware problem in cloud between cellsites. North and South have same infrastructure and software.

12:39: Can't give absolute deadline when all cellsites fixed; no timeline given.

12.38: Problem is in transport layer between cellsites; Quite a different error to the one on December 14. Trying to take actions so not repeated.

12.37: Percentage of calls completed is now in mid-ninties.

12.36: Current state in last hour: number of unrestored cellsite now down to "mid-30s"; being restored slowly so comeback is sustainable, doesn't affect rest of network.

Latest XT Mobile Network Situation Update

(Sent by Gen-i chief executive Chris Quin to clients around 11.30pm last night):

This is a further email to update you on the restoration of services for XT mobile users between Taupo and the bottom of the South Island in the last two days, and an update on the actions we are taking.

I am conscious that to date we have been giving you updates on the status of the outage, and I now want to start explaining the issues and to set out some of our commitments going forward.

To give an overview, we have a total of 986 cell sites nationwide, of which 453 are linked to our Christchurch XT Mobile RNC switch.

Since 10.30am yesterday (27 January), at times up to 54 cell sites have been out of service, and XT users will not have had access to voice, SMS or mobile broadband services. At the time of writing this update (approx 11.30pm, 28 January) we had successfully reduced the number of sites out of service to 38 and further work continues through the night which we will update in the morning.

The full list of impacted cell sites is included below. They are a small set of cell sites in the area from Taupo south, the majority in the lower South Island, including Dunedin, Invercargill, Timaru and Queenstown, as well as some parts of Wellington central, Taranaki and Ruapehu.

While the remaining 94% of our cell sites nationally have not been affected, traffic loading on the network may mean that a percentage of calls in the Taupo south area may not get through. 

Our average statistics for call accessibility in the southern region and retention in the area have been above 90% in the last while.

Our absolute focus right now remains on restoring service in all these locations, and ensuring that restoration is sustained and that we deliver a stable and world class XT network as we know we can.

The intense level of focus and monitoring from the last few days will continue. We will continue our optimisation of XT to ensure the coverage, speed and capability is world class.

The root cause is now under extensive investigation, but is suspected to be within the physical and logical paths in the transport layer between the cell site and the Christchurch switch. This is a different issue to the one that caused a disruption of services in December. The technical teams are working through a range of possible causes for the failure.

The remedial action over the last two days has included:

- Rewrite of the MIB which requires an RNC reset to install.
- A reset of the UTRAN router to clear a path.
- Reset an interface.
- Lockout of the base stations connected to suspect links.
- Managing the registration “storms” resulting from mobile re-starts from these base stations.

Further work that will occur overnight includes analysis of traces on the suspect links to clear them so base stations can be unlocked and returned to service.

We cannot currently accurately forecast a time of full and final restore.

I wanted to comment on the questions I have received about the redundancy of the network and issues like the number of “RNC’s”.

Design of resiliency and redundancy of a mobile network is not only to do with RNC’s. In fact within the UTRAN there are several factors including the number of RNC’s, routers, TMU’s, links as well as the number of users and type of devices in use.

We have always expected we would increase the capacity of the XT network as we add more clients on the XT network, and we have seen a large number of clients move to XT in a relatively short space of time. However, more RNCs alone do not deliver greater redundancy – each RNC is equipped with its own hardware redundancy but more RNC’s will increase resiliency as we spread the load and that’s our plan.

The Telecom Tier 2 and 3 teams are currently heavily engaged with Alcatel-Lucent, including “24/7” support capability from China, Singapore, Germany, Netherlands, USA, Canada, and France. In addition, Alcatel-Lucent management at the highest level from around the world are engaged on this issue locally and in France, including the CEO of Alcatel-Lucent.

Our Gen-i client teams are continuing to work directly with affected clients today to support their business needs. Our immediate focus is on service restoration and we will look closely at claims for compensation once the service is restored.

We will also continue the open and honest communication on the event and our actions as we go forward. Tomorrow (Friday 29th January) at 12.30pm I will be leading a webcast for all our clients, and invitations to join me will be sent to you in the morning. I hope to see you there, and answer any questions you may have, and provide the latest update personally.

I am continuing to work with my Telecom Exec colleagues and our clients to resolve the issues and deliver the actions going forward. Once again, we apologise and deeply regret any inconvenience or problems you are experiencing due to the interruption of the XT service.

Alongside a design review by the R & D teams, Paul Reynolds has commissioned an urgent and independent review to ensure we have taken all steps to assure our XT client experience. Any feedback you have provided to me this week will be collated and added to that review.

In addition, we will be seeking your input to understand what further requirements will best meet your needs on the XT network. We will openly share the recommendations of the review with you.

Finally, I have been keen to communicate to all our clients and all New Zealanders on this issue, including those that are not using the XT mobile service. While the impact is not direct for other network users, a key part of why we update executive level clients whenever we have issues like this is to ensure that people trying to contact affected XT clients are aware of the issue. And it’s important to us that this issue is communicated directly by us.

We will continue to equip our client teams with information on the event and future actions. If you want to discuss this further with me, or any of the Gen-i team, please let us know.

Affected sties, as of 11.30pm last night:

Site name	Location
St Clair	Dunedin City
Clinton	Clutha
Clyde Cell Site	Central Otago
Brighton	Dunedin City
New hospital	Invercargill
Carisbrook	Dunedin City
Invercargill State Ins.	Invercargill
Corstophine Radio	Dunedin City
Peninsula Hill	Queenstown
Gleniti	Timaru
Glencoe	Queenstown
Winton	Southland
Grantlea	Timaru
Mt Alford	Ashburton
Waikiwi	Invercargill
Surrey	Invercargill
Ardlui	Dunedin City
Cantec	Timaru
Signal Hill	Dunedin City
Oamaru	Waitaki
Dunedin Oval	Dunedin City
Mosgiel	Dunedin City
Majestic Tower	Wellington
Tennyson street	Dunedin City
Waldronville	Dunedin City
Mt Horrible	Timaru
Camp street	Queenstown
Swampy summit	Dunedin City
Timaru South	Timaru
Lake Hawea	Otago
Halfway Bush	Otago
Te Anau	Southland
Luggate	Otago
Otago Polytech	Otago
Waimate	Canterbury
Brothers Peak	Otago
Millers Flat	Otago
Waitati	Dunedin