Member log in

Dick remembers the day the Southern Cross Cable lost service - and says it will fail again

In the wake of Pacific Fibre’s funding failure, the heads of major ISPs are worried about threats to the 50% Telecom-owned Southern Cross Cable - New Zealand’s only high capacity connection to the outside world.

Concerns include human error, slicing anchors, acts of God, island dictatorships and a clocking ticking on mean time before failure.

Orcon boss Scott Bartlett told NBR he worried about an earthquake hitting the Southern Cross Cable's landing points in Auckland - an event he said could leave the country without internet for weeks or months.

"Bartlett’s mischievous assertion of the “Southern Cross Network as a flimsy piece of infrastructure” is completely wrong," Southern Cross director of sales and marketing Ross Pfeffer told NBR.

"The Southern Cross Network* comprises two fully diverse cables, using two geographically separate cable stations in each country: NZ, Australia and Hawaii and the US West Coast.

"The network is configured as a self-healing figure “8” and is designed to achieve better than 99.999% availability, equivalent to 5.25 minutes of unavailability annually. Over the last 7 years customer circuit availability has exceeded these design figures.

"To minimise the potential for dual failure in the event of disruptive seismic activity our NZ cable stations have been placed 10.5 kilometres apart at Takapuna and Whenuapai with cable landings on the east and west coasts. [see diagram above right]."

As NBR has previously reported, the figure 8 design allowed the cable to survive an undersea earthquake that took out a shunt on one of its two loops (which also make the 26,000km cable more expensive to maintain).

The day Southern Cross went down
It's a neat argument. But no system is infallible.

And indeed CallPlus chairman Malcolm Dick points out that the Southern Cross Cable suffered a major outage over July 28 and July 29, 2001 (in online comments, Dick as been going on memory, but a Southern Cross incident report obtained by NBR closely aligns his version of events).

LATEST: Southern Cross Cable responds to Dick attack over 20-hour outage

A freak series of events saw both cables in the Southern Cross Network out of action - one down for routine maintenance, the other sliced by a ship's anchor in Sydney Harbour (where a had hit storm and ships were warned to weigh anchor downwind of the cable. One vessel had two anchors down. A crew member only pulled up one. The other dragged through the cable). 

Software used to manage failovers failed.

"This caused a total outage, and Southern Cross staff had to manually route the working segments together, creating a 14-hour outage," Dick says.

Some customers were left without service for 12 hours, others for up to 20 hours.

"Fortunately at that time it was still the dial-up era and there was not much traffic on it, so it attracted little press," Dick told NBR.

The incident also highlighted that while Southern Cross offers fully redundant or "protected" service, not all telco and ISP customers stump up the extra cost required to take advantage of it.

Likelihood of future failure
Dick has also been looking at the statistical likelihood of of a future failure - and he doesn't like what he sees.

He details:

Looking at industry statistics for cable failures:

  • Terrestrial cables have an average failure rate of 1-2 failures per 1000 years per kilometre
  • In various studies I have seen the comment that Submarine cables are about 10 times more reliable that Terrestrial cables
  • Terrestrial cables have a repair time generally of < 48 hours
  • The Southern Cross cable has a repair time (estimated by Southern Cross execs to me some years ago) of three weeks

The maths says we can expect a total outage about once every 15 years," Dick says.

"The outage could happen tomorrow.

"[That's] frightening when you consider our newfound reliance on broadband and the fact that a break can take up to three weeks to repair."

The failure could happen in years, or it could happen tomorrow, Dick says. Either way it will be "catastrophic."  

Dick backs comments from Pacific Fibre cofounder Rod Drury that a second cable operator was need not just for price, but redundancy and security.

Satellites there for fall-back, but would be overwhelmed
Snap Internet CEO Mark Petrie has been one of the few to publicly back Southern Cross on pricing - an area where he has faith the single operator will do the right thing.

But in term's of the possibility of service failure "I second Malcolm Dick's comments" the Snap boss told NBR.

"It is quite a frightening prospect. There would be no option for satellite capacity in my opinion. Maybe for essential services, but residential broadband would come to a very quick halt."

Fiji threat
Dick sees more threats.

"The cable runs into and past Suva in Fiji which is now a dictatorship," he told NBR.

"A couple of years ago, Fiji had a dispute with Tonga resulting in the Fijian Navy destroying the lighthouse put up [on disputed Minerva Reef] for safety."

Tonga erected two new lighthouses to guard the reef.

Dick was sailing through Minerva Reef a couple of weeks ago, and took the photo above of a Tongan naval vessel on patrol.

There are broader tensions.

"The possibility exists that if we annoyed Fiji too much, they have the capability of decommissioning one segment of the cable, leaving New Zealand with a single cable, virtually guaranteeing outages from time to time," Dick says.

Southern Cross has not so far responded to a request for comment on Dick's mean time before failure calculations.

* The cable's full name continues to be The Southern Cross Cable Network; Mr Pfeffer appears to have taken to have switched to calling it the Southern Cross Network to emphasise it has two strands of cable.

By the industry veteran's calculations, we can expect a total failure once every 15 years.

The failure could happen in years, or it could happen tomorrow, Dick says. Either way it will be catastrophic.  

By the industry veteran's calculations, we can expect a total failure once every 15 years.

The failure could happen in years, or it could happen tomorrow, Dick says. Either way it will be catastrophic.  

By the industry veteran's calculations, we can expect a total failure once every 15 years.

The failure could happen in years, or it could happen tomorrow, Dick says. Either way it will be catastrophic.  

 

More by this author

Comments and questions
23

Southern Cross claim 5.25 minutes downtime a year, but already has had a 14 hour outage? That means it shouldn't have another outage for 160 years!
Easy to work out who is bullsh*tting

Need to read the article carefully, and the incident report that Chris attached. Yes SX says 5.25 minutes is the statistical average for a failure of both cable paths, and you can see that there was an icident that lasted 14 (to 20 hours), but as you can see in the incident report neither the traffic from NZ to the US or to Australia was affected which is utlimately the point of what SX are saying.

So I agree its easy to see ... but doesnt look like its necessary SX which Chris and others seem to be pushing. Makes good copy though!

How many people will switch off Xero in the event of a catstrophic failure of the cable. Lots I reckon. Mr Drury must be a teeny weeny bit concerned about that....

Not many actually, as Xero use Akamai, who route around damage - you connect to one of the half dozen akamai nodes in New Zealand.... Akamai's network is designed to flip over around damage in minutes - as long as aussie + us routes were not all out...

I could be wrong but I don't think Akamai would help in that case, as what a Xero customer would be concerned about is getting access to the database, not the website images and other assets

Possible but a lot scare mongering.
Why not put the money where your mouth is - which is why the new venture collapased it wasnt visionary it was a fight to take on the incumbant and pure ambition. They had plenty of time to cobble up a investors but couldnt.
Tall poppy syndrome is rife in NZ.

I also thinks its somewhat disingenious for Orcon to lament the demise of PF, and critic the reliance on SX etc, but in the same breath admit that they didn't invest a single dollar into PF.

Come on guys, if the ISPs who say they want/need/must have additional cables (for whatever reason) are not willing to put money in, and the rich list are unwilling to put their own money in, its no wonder they cant get another off the ground.

Scott should worry more about Orcon issues. Auckland is outside the earthquake zone. There are no 100% guarantees with technology, even banking systems. Put some perspective into NBR rather than acting as posthumous PR agents for Pacific Fibre.

Someone else will come along, understand the issues better, finance it and build it

Are you kidding me? We weren't expecting Christchurch (it was a hidden fault until it happened), so it would be silly not to expect and prepared ourselves in case something happens to Auckland.

Auckland is built on a bunch of volcanoes. Sure, just because they're sitting mostly dormant now, the unpredictable nature of the earth's crust means anything won't happen tomorrow. Or 10 years from now. Or 100 years from now. or 1000 years from now.

For the sake of ensuring continuity and preparing the country for the worst, adding a few hundred million to the already existing UFB build cost probably isn't a silly idea but, also to actually, you know, allow the UFB to deliver on it's promises of 100mbit/s for all: no point in having that kind of pipe if you can only get those speeds domestically.

SxC is already dangerously over-utilized on it's current equipment and frankly, getting a bit grey these days - it's well over half way in to it's service life and we need to look at supplementing (and replacing) the cable immediately, whether publicly or privately funded (or both), as any cable started now is going to take the better part of 3 years - maybe more - to complete, by which time SxC will be not too far from retirement.

If I had half-a-billion spare I'd build it but sadly as it's not the case all I can do is champion the idea to people who do. Granted, I have to wonder perhaps why the Pacific Fibre guys didn't self-fund: I'm sure they have a combined net worth of well over half a billion, so they wouldn't be going "all in", but even then, of the few NZ billionaires surely a couple would benefit significantly from such an enterprise based on their business interests?

In all this rhetoric people keep forgetting that only one NZ company has actually put the money to fund a cable so that you can all watch porn and youtube...

So while TNZ makes a dividend (i seem to remember it being $50m pa?) it also put up the original money to fund it....

Drury tweeted "today i learnt that hardware is harder than software"... which shows he didn't learn butkiss, this is an infrastructure game...
Do even the most basic back of the fag packet analysis... so TNZ gets $50m, Rod etal wanted $400m, WACC of generously 5% - $20m just to cover costs... how hard do you think it would be for SX to price match and ruin that business case...

I'm surprised the Scumatawhenawa haven't tried to claim internet data yet.

shhh- part of it crosses the foreshore.

When Egypt's internet went down for 5 days in 2008, OECD worked out it cost them $110million.

http://www.forbes.com/sites/parmyolson/2011/02/03/how-much-did-five-days-of-no-internet-cost-egypt/

To be fair the folk at Southern Cross do have a heart. A couple of years ago a USA landlink was damaged by flooding in Oregon and was out for days. Those ISPs who had purchased only unprotected capacity on the system to save a buck were also restored on some spare capacity even though Southern Cross had no obligation to do that.

Is it time now for some discussion on how we can find a solution that merges decent physical diversity, commercial competition, and shared construction cost into an innovative solution? It might even include Southern Cross but whatever it should not have NZ being connected only to Australia. I think that would be short sighted for NZ business.

Sounds like yet another case when we were down to one link for a few days

if the govt cared a damn about anything other then being re-elected, they'd fund a second cable, but nope not a bean.

Perhaps if Southern Cross fails and the cyber apocylpse means we can't all hang out online we might do something novel like ressurect the art of talking, communicating and experiencing the now... would that be such a sad thing?

The Fiji risk argument makes it difficult to take anything Malcolm Dick says seriously

" If we annoyed Fiji too much "

Who is " We " I would ask ?

Southern Cross Cables is a company registered in Bermuda.It is neither NZ owned nor NZ controlled .

Fiji has just the one cable connection at the moment.

The idea that Fiji would cut their only cable link with the rest of the world just because NZ annoyed them, is the thought of someone struggling for a good argument but failing

They could cut our cable to the USA and still be connected to the USA themselves. Look at the network map.

yes they could,I may fly over on a pig and show them how !!!Duh

Fiji are trying to position themselves as an island hub for internet .... with the new Tonga cable connecting there and recent announcement about a Vanuatu cable ... Can't see how even the most power hungry dictator would think its a good idea to shoot his own foot off by ticking off the company that connects them to the US!

Bula for them LOL!

Some years ago, after a BBN packet switch install for an NZ subsidiary of Barclays bank, I made a suggestion what if Barclays (not now!) were to fund an infrastructure build to create a Pacific Switching and Relay Centre for the then data requirements for Pacific Rim nations. The aim was to subsidise NZs cable costs by providing the Hosting & Network Management of this infrastructure resource within the globally strategic safe-haven country of NZ (ie a trusted but remote backwater of the world). We would be able to act as a backup Pacific switching node for the major economies of the Pacific during outages being the small, educated and trusted 'fair dealer' of our region...Barclays at the time, managed their global network at three points London + 2 other locations separated by 8 Hr time zones. Network Managed 'roaming' if you will. Given our rapidly increasing reliance on delicate (political) & physical communications infrastructure - could a working business case be proposed to support such a win-win option? Security of Undersea Cable-Network Operations for PACRIM economies vs Highly skilled, highly paid careers, plus critical mass for a Weightless export economy for NZ.