[UPDATE: at 12.55pm Telecom said all affected cellsites were restored and running as normal. The company, whose first communication to corporate customers said 126 XT sites were affected, has now revised the number upward to 200.The latest corporate clien
[UPDATE: at 12.55pm Telecom said all affected cellsites were restored and running as normal. The company, whose first communication to corporate customers said 126 XT sites were affected, has now revised the number upward to 200.
The latest corporate client update, not yet made public, says ""The issue is believed to be due to a variable level of service performance from the network that supports cell sites attached to one of our Auckland RNCs".
See the second Gen-i update at the end of this story for Telecom's full official explanation.]
After a morning of intermittent faults, Telecom has acknowledged that its XT network is having problems in Auckland.
Customer complaints first hit Twitter at 9.30am this morning. Northland XT subscriber Kirsty Hughes told NBR she could not connect to the network from that time - with her iPhone tethered to a laptop - and received no response from Telecom. By midday, here phone was working again.
On Geekzone, members have been complaining about dropped calls, failed connections and txt delays. Glenfield and Albany were among suburbs hit.
"It [XT] is not completely down, but there are some issues," Telecom spokesman Ian Bonnar told NBR.
Mr Bonnar had no immediate comment on the cause of the XT problems.
126 cellsites affected
In an update emailed to Telecom corporate clients at around 10.18am this morning (see full text end of story), Geni-i boss Chris Quin said the problems affected 126 cellsites in Auckland and Northland.
The number of cellsites affected had been contained by recent efforts to make XT more resilient, said Mr Quin.
Second Auckland cotroller 75% complete
Telecom is in the process of installing a second RNC (radio network controller) in Auckland.
In a separate email to large Telecom customers, sent yesterday, Mr Quin wrote.
"Our programme to migrate cell sites to our new (fourth) Auckland Radio Network Controller (RNC) is 75% complete, with 155 cell sites successfully migrated (out of a total of 202).
"We plan to migrate the remaining cell sites to this RNC by the end of the week, and will rebalance all four RNCs by mid April, giving us increased capacity and resilience."
The first four XT outages, in December, January and February, all hit areas Taupo South and were all, in some way, related to Telecom's Christchurch-based RNC.
As of December, Telecom had two RNCs, one based in Christchurch, controlling all XT network traffic Taupo south, and one in Auckland controlling traffic Taupo north.
In his email today, Mr Quin said there were issues with the network that sits between 126 cellsites and Auckland RNC1, rather than issues with the controller itself.
"Cell sites serviced by the two Christchurch-based RNCs, and the second Auckland-based RNC, are unaffected," wrote Mr Quin.
RAW DATA 2: Gen-i's 2.21pm update to clients
Subject: XT Mobile Network Update - Thursday 8 April
Subject: Latest XT Mobile Network Situation Update - 14:21 hrs 8 April 2010
Dear [snip]
As I have communicated to you in my note earlier today, some XT clients in Northland and the Auckland region north of the Bombay Hills may have experienced intermittent issues with XT mobile network voice, data and text services between around 9am and 12pm today.
I can confirm that the affected cell sites in the region have been fully restored, and that the network is now running as normal.
We can confirm that this issue was not related to the performance of either of our Auckland Radio Network Controllers (RNCs). The issue is believed to be due to a variable level of service performance from the network that supports cell sites attached to one of our Auckland RNCs.
We located the cause of the issue in the network and invoked our fail-over mechanisms for these systems. This worked as planned, and successfully restored all services.
At the height of issue, up to 200 cell sites were impacted. Cell sites serviced by the two Christchurch-based RNCs, and the second Auckland-based RNC, were unaffected.
111 calls were monitored during that time.
As for any mobile network service, there are occasionally unexpected issues that lead to a degraded service. This was not an outage – it was reduced accessibility of mobile services on part of the network for a short period of time.
We will continue to keep you informed on the programme of work to improve the XT experience for our clients.
In the meantime, please contact me or your Gen-i client manager if you need further information.
Chris Quin
CEO Gen-i Australasia
M: 027 442 0902
RAW DATA I: Gen-i's 10.18am update to clients
Subject: XT Mobile Network Update - Thursday 8 April
Dear [snip],
I’ve committed to keep you informed on the XT network experience going forward, and ensure we keep being straight up on any service degradation with XT.
From 9am this morning (Thursday 8 April), some XT clients in Northland and the Auckland region north of the Bombay Hills may be experiencing degraded service with our XT mobile network voice, data and text services. The impact will lead to difficulty at times in accessing these services if an XT user is working in one of the affected cell site areas.
We can confirm that this issue is not related to the performance of either of our Auckland Radio Network Controllers. It is believed to be due to a variable level of service performance from the network that supports cell sites attached to Auckland RNC1. Investigations are being carried out across all components of the network to locate the cause of the issue, and as at 11.30am, one particular part of the network was being focused on as the potential cause.
Due to our ongoing programme of work to improve XT mobile network coverage, capacity and resilience, the impact has been contained to up to 126 cell sites. Cell sites serviced by the two Christchurch-based RNCs, and the second Auckland-based RNC, are unaffected.
We have implemented our emergency services contingency plan. 111 calls are being monitored, however, we advise any clients having issues calling 111 from an XT mobile to immediately use a landline.
We are working with our network partner, Alcatel-Lucent, to take immediate steps to restore service, and will give you updates as the situation develops.
In the meantime, please contact me or your Gen-i client manager if you need further information.
Chris Quin
CEO Gen-i Australasia
