Member log in

Curran pounces as fault hits Southern Cross Cable

A ‘catastrophic failure’ has struck the Southern Cross Cable, claims Labour ICT spokeswoman Clare Curran.

The 50% Telecom-owned company, which operates New Zealand's only major broadband link with the outside world, confirmed the problem, but said it only affected 10% of its capacity. The Labour MP was being misleading and inaccurate, the company says.

“Labour has learnt that a ‘catastrophic failure’ at Southern Cross’s Alexandria [New South Wales] landing station occurred this morning due to an unauthorised and un-notified software change to their wavelength switching platform, which blew up," Ms Curran says

The Dunedin South MP told NBR her source was a network operator, from outside New Zealand, and forwarded part of an email concerning the incident which calls Southern Cross' lack of an alert "unacceptable".

The incident, although apparently minor, highlights that price competition isn't the only issue in the second-cable debate. 

Kordia NZ/Orcon boss: highlights vulnerability
Kordia NZ CEO Scott Bartlett confirmed the outage, telling NBR "It lasted approximately one hour and 10 minutes. Unprotected capacity dropped between New Zealand and Australia and Australia and the USA  – very serious. Orcon not affected as we have capacity going in both directions.  The larger impact would have been on Australia, not NZ."

He added, "I have been preaching that we need another cable system for this country for a long time. Technology fails, its a matter of when and by how much, not if."

Snap boss: not a catastrophe 
Snap CEO Mark Petrie, who happened to be at Southern Cross' Alexandria Data Centre this afternoon, told NBR the cable operator had three fibre pairs crossing the Tasman, going into three different bits of equipment. On pair failed. Snap noticed no difference in performance.

Switched fibres, three pairs into three bits of equipment, just one bit of equipment, no loss of service to Australia to not redundancy

"Another cable an excellent idea from a commercial point of view and more redundancy would be better," Mr Petrie said.

But in technical terms, his company suffered no problem today.

Earlier, CallPlus chairman Malcolm Dick recalled a major Southern Cross outage in 2001 - and highlighted the fact that although the cable operator offers "protected" capacity, thanks to its twin cable design, not ever customer shells out extra money to take advantage of this redundancy.

The government supported start-up Pacific Fibre with a major contract to buy bandwidth for the Crown-owned Reannz network. But the cable's backers argued the Crown could have done more, including making it mandatory for UFB companies to buy international bandwidth from more than one provider, or encouraging the $20 billion Super Fund to invest (the fund called the cable too risky).

Pacific Fibre co-founder Rod Drury recently began a push for a public-private partnership model to fund an international cable, formally putting the idea to ICT Minister Amy Adams.

Major outage in 2001
In August, CallPlus chairman Malcolm Dick related to NBR a major outage on the Southern Cross Cable in 2001.

A freak series of events saw both cables in the Southern Cross Network's twin-cable design taken out of action - one down for routine maintenance, the other sliced by a ship's anchor in Sydney Harbour (where a had hit storm and ships were warned to weigh anchor downwind of the cable. One vessel had two anchors down. A crew member only pulled up one. The other dragged through the cable).

Software used to manage failovers failed.

"This caused a total outage, and Southern Cross staff had to manually route the working segments together, creating a 14-hour outage," Mr Dick said.

Some customers were left without service for 12 hours, others for up to 20 hours.

"Fortunately at that time it was still the dial-up era and there was not much traffic on it, so it attracted little press," Mr Dick told NBR.

The incident also highlighted that while Southern Cross offers fully redundant or "protected" service, not all telco and ISP customers stump up the extra cost required to take advantage of it.

Mr Bartlett backed Mr Dick's comments, saying a single cable operator made New Zealand more vulnerable in the event of a natural disaster hitting Auckland, where the Southern Cross Cable lands (at twin points on each coast at Takapuna and Whenuapai). The country was at economic risk.

Southern Cross told NBR the 2001 incident happened at a time when the Hawaii to Oregon segment of the cable was not complete.

It could not happen today with the full, twin-cable design in place (and indeed, while one of the cables might have been taken out this morning, NBR at least noticed no international connection problems).

One cable down, the other overwhelmed?
Snap CEO Mark Petrie accepted the twin cable design meant a total outage was unlikely, but feared the remaining cable could be overwhelmed. He adds that, regardless, two cables are better than one in terms of price competition.

Fiji threat
CallPlus' chairman sees threats beyond technical failure.

"The cable runs into and past Suva in Fiji which is now a dictatorship," he told NBR.

"A couple of years ago, Fiji had a dispute with Tonga resulting in the Fijian Navy destroying the lighthouse put up [on disputed Minerva Reef] for safety."

Tonga erected two new lighthouses to guard the reef.

Dick was sailing through Minerva Reef mid-year, and took the photo above of a Tongan naval vessel on patrol.

There are broader tensions.

"The possibility exists that if we annoyed Fiji too much, they have the capability of decommissioning one segment of the cable, leaving New Zealand with a single cable, virtually guaranteeing outages from time to time," Mr Dick said.

ckeall@nbr.co.nz


RAW DATA: Southern Cross Cable statement

Contrary to a misleading and inaccurate media release from Labour’s Clare Curran, no ‘catastrophic failure’

has occurred on the Southern Cross Cable.

The cable is, a figure of 8 network providing internet services to New Zealand, Australia, Pacific and the US.

In the early hours of this morning a limited outage affecting 10% of our active capacity occurred during our
maintenance window which is a low traffic impacting period.

The outage occurred at one of our Sydney cable stations, Alexandria, and it lasted from 3.17am – 4.28am, Sydney Time, impacting 4 of our customers*.

A problem occurred and the switch was reverted to its original software.

The incident occurred as a part of authorised work taking place to expand capacity on the Southern Cross network.

NBR understands one of the customers was Southern Cross wholesaler Vocus, which has previously told NBR it onsells access to the cable to around 25% of the NZ retail ISP market. 

On this point Southern Cross replied: "The 4 customers were either ISP or telecom companies. We do not comment on who our customers are. It is hard to determine the NZ impact but [it] was 10% of our total traffic."

More by Chris Keall

Comments and questions
24

Alternative sensationalist headline: Mega failure on Southern Cross cable

Labour’s Communications and IT spokesperson using "blew up" to describe a software fault.

Why is this woman in this job when it's clearly over her head ??!!

Did anyone (outside the ISP's) actually notice this 'catastrophic' failure. Or did the built in redundancy of the figure of eight design work as planned?

Over 10 years since the last reported failure and it was fixed in just over an hour.

Yep, I certainly noticed when I couldn't access overseas websites early this morning. Took at least half an hour for my ISP to transfer over and restore connectivity.

Yep, the updated BGP4 vector path tables took a while to a while to propagate from us to retail, sorry!

This issue definitely affected networks our backup units in Melbourne had huge issues connecting to our other networks, this was resolved as we used our secondary upstream provider. Don't think NZ was affected as much but you'll notice issues when connection to networks in Australia.

The last outage as mentioned above was around 14 hours, not 1 hour...

The figure-of-eight design became a Jesus fish, with the big gap being at Alexandria. There was a huge impact on people downstream.

Agree - atleast she's "reasonably credible" by comparrison to her colleagues...

This is another attempt by Labour & the unions to manufacture a "crisis" where none exist (or very little) - except with a willing, shrilling MSM politically sympathetic churnalist – maybe with aspirations to join the Labour Party?

This is almost as transparent as the union-planted EPMU shriller who features in the Horrid today as a single mum who can’t find work etc etc… her innocence sits right alongside her also being a Member of the Labour Party Waitakere Electorate Committee, and a member of the Executive Committee of the EPMU – thanks to Whaleoil for highlighting that fact.

There’s the exact same story running in the Christchurch Press today – similar story, different gender… and I wonder which union / Labour Party committee he belongs to?

Reminds me very much of the tales of woe MUNZ tried to paint about their member Mr. Cecil Walker as they held the Ports to ransom trying to extort 1950’s work practises on them… and we all now know how much lies and deceit MUNZ are prepared to spout if it fits their propaganda agenda…

The more cables the merrier.

Provided we don't have to pay for them - otherwise as many cables as the market can get a return from ... which appears to be one.

Reply to #1.

Because she was quoting the email from [company name, and email header with addresses included, but deleted before this comment was published - CK]

All,

We were operating under a hazard for reduced protection. I've just finished on a conference call with SouthernCross who have admitted this morning they performed an unauthorised and un-notified software change to their wavelength switching platform at Alexandria, this blew up. Shortly after this we were able to bring the protection path back into service, restoring all services. As I understand it from SouthernCross the software upgrade issue is still ongoing at Alexandria, SouthernCross are working with the vendor to fix it / rebuild it but currently they are operating on a single path.

A new hazard notification is going out to customers about the lack of current protection and continuing issues SX are having at Alexandria.

We are sorry for the inconvenience caused and I've escalated the lack of notification within SX as it is simply unacceptable.

If SX say the fault occured at 3:17am Sydney time, why I am seeing the original fault report on AusNOG at 1:24am Sydney time?

Be aware that SX is operating a SINGLE PATH currently. If anything should happen to that path, there will be a total outage.

I'm not a fan of SCC's monopoly but for a 'catastrophic failure' this seems fairly benign.

The presence of an alternative cable just means you either buy the protected service on SCC or you buy capacity on both. Either way you're spending more than the unprotected circuit that those affected forked out for.

Given PF was going to land in Auckland the country would be no less vulnerable if a natural disaster struck Auckland had that cable got off the ground.

Another diversionary non-story whilst Labour wannabees organize the next BBQ to plot against their leader, David "the lamb" Shearer......

Don't forget the great rat chewing through cable incident
21st June, 2005 : A communications problem occurred in New Zealand after a rat chewed through an important fibre optics cable shutting down the country's internet and mobile phone services for about five hours. The problems were exacerbated when a workman in another part of the country damaged another main cable by accident at the same time as the rat incident.

Is there anyone with a brain in the labor party?

Its great that the redundancy has kicked in and we are all still operational.
I just hope the Southern Cross crew are out in the Hauraki Gulf making sure that no-one's anchored off Takapuna over the cable path, because that's the only one we have left..

lol... not mention about Kim Dotcoms support for a second cable.

Good keep go curran - wouldn't it be great if it broke..that means people would have to talk to each other and even pick up and the phone to chat. Think about all the students will have time to study instead of texting. There are so many human upsides of it falling over..even the greens would agree.

"There are so many human upsides of it falling over.." Well and truly trumped by how much emergency,medical, biz, govt (screw it all society) services increasingly and massively rely on broadband connectivity.

International connectivity is not required for txting (unless it's international txting obv).

Do you work at being an idjit or just it come naturally?

One provider is outrageously risky for an entire country to depend on.

I seem to remember another 'minor cable glitch' at the turn of the century that made NZ the laughing stock of the world.
When do me learn? Do we ever learn? ....sounds a bit like a song I once heard. Well almost.....

I run a webhosting business that offers AU hosting to NZ clients.
Of all my clients that rent this service, only those who have broadband with ICONZ suffered - they couldn't access their website and instead received a cached version from the time of the incident for around 12 hours.

ICONZ seems to have come off the worst in this incident.

This is unnacceptable the SNAP can endorse unplanned maintenance (software update). Heads should role for allowing "unauthorised" access.

Clearly NZ need an additional pipe for resilience.
On a lighter note, I bet Kim Dotcom is happy with this event since he is planning a "pacific fibre pool party" this week. ! :-)

This just goes to show that if one cable can go down then two can as well. I agree we'd all just have to all to each other more, but sadly today we rely on international comms for so much. For example our exchange rte mechanism relies on frequent automated reporting, our banking systems route a massive number of transactions overseas and airline check-ins worldwide bound for NZ would suffer huge delays. With KD and RD talking separately of new cables then let's at least bring them ashore somewhere away from Auckland to boost our resilience. As KD has blogged, we could build a strong Internet economy here as well if we had better cable infrastructure, capacity and price. New cable(s) could give us all of these things.