Xero taken offline by massive US data centre failure

One of the drawbacks of cloud computing was dramatically illustrated today as Rackspace - one of the world's largest web hosts - when offline for 45 minutes. New Zealand’s Xero was one of many SaaS (software-as-a-service) providers knocked out by the failure, with glitches continuing for hours.

The accounting software provider went offline around 8.30am as Rackspace, which hosts all of Xero's data, was hit by a still-unexplained, catastrophic failure.

All Xero servers were back up and running by 9.10am chief operations officer Alistair Grigg told NBR.

Some customers were still reporting problems logging on through the morning and early afternoon, as recorded on Xero’s blog. Mr Grigg says these were cookie and DNS (domain name server) issues, which were resolved by asking customers to restart.

SaaS, interrupted
Rackspace’s catastrophic outage has shocked the computer community worldwide.

The fault was caused by a power failure at the US company’s giant data centre in Dallas.

But with Rackspace maintaining server farms around nine locations in the US, UK and Hong Kong, it’s not clear why a failure at one facility took its systems completely offline.

Mr Grigg says Rackspace has multiple utility providers, backup generators and uninterruptable power supply systems at its Dallas facility, as would be expected from one of the world’s largest data hosts.

So far, no official explanation has been forthcoming.

The power fault also took out Rackspace's own website and help centre, adding to the confusion. It was left to the company's Twitter account to relay the disaster to the outside world.

When Rackspace came back online, it was running on a mix of utility and backup power, Mr Grigg notes.

The COO speculates that “there must have been some pretty significant component failure possibly at the point where maintenance work was being done”.

The outage marks Xero’s first ever hosting failure during its first two and a half years of operation. Mr Grigg says only a handful of UK and Australian customers were affected, due to the timing of the outage.

Has his faith in the cloud been shaken?

No. Mr Grigg says that after 27 months, Xero still has a record of more than 99.99% uptime.

He said the company could look at a second cloud host, but that such a disaster recovery switch typically took several hours, whereas Rackspace got Xero back online within 45 minutes (some customer with more complex systems took longer).

Xero customers could take some comfort in they were among millions hits - Rackspace hosts sites and services for more than 62,000 companies - with US singer Justin Timberlake leading complaints via, what else, Twitter.

This article is tagged with the following keywords. Find out more about My Tags

Post Comment

13 Comments & Questions

Commenter icon key: Subscriber Verified

Really disappointed with Rackspace. We pay a premium for their service, not because we expect perfection, but because we want 'the best' when something does go wrong.

Once our sites went down, we couldn't get through to Rackspace on the phone. Their online support system was unavailable. And when we finally did get through received no information at all. Their twitter account was even slow to post updates.

After their incident in November 2007 (truck crash took out our servers) and then reading about reported incidents from other companies earlier this year, I've lost faith. It does not seem acceptable that their backup systems have failed on multiple occasions - it makes one wonder about whether they promise more than they can deliver.

We are not a huge customer, but we are not small either - around $100K/yr. We are going to look for alternatives, and if we do indeed move on, I'm skeptical that our account rep will even notice or care. This just may be a case of growing big and indifferent. It's a shame.

Reply
Share

We work with several data centers throughout the US that have tested and proven back up power and generating equipment. Testing is required on a regular basis to ensure that when needed.....IT WORKS! If you are looking for alternatives, call me at 817-529-9675.

Reply
Share

I was down for 5 hours...

Reply
Share

This just goes to show, no matter whether your accounting package is locally hosted or in the cloud, we will always be susceptible crashes. I guess in defence of the cloud option, at least we did not have to pay for an IT consultant to come round and fix the problem.

Reply
Share

Puts me in mind of the comedian Louis CK's recent rant on Conan O'Brian, namely the fact that everything is amazing and nobody is happy. You know what? Servers go down, power goes out and accidents happen. Yes it was a huge inconvenience for many people and made business difficult for many others as well, but when you consider this was a hiccup in an industry less than 5 years old for a service that performs services at 50 times the speed on which we were forced to wait previously maybe the best solution would have been to go out for a long lunch rather than having a hypertensive stroke over being denied a service people hadn't even imagined as a possibility a decade ago.
<a href="http://www.sudlows.com/">Sudlows</a>

Reply
Share

Great blog! I genuinely love how it is easy on my eyes as well as the info are well written. I am wondering how I may be notified whenever a new post has been made. I have subscribed to your rss feed which need to do the trick! Have a nice day! <a href="http://www.coppervalleycustomhomes.com">custom home builders</a>

Reply
Share

Hello,I love reading through your blog, I wanted to leave a little comment to support you and wish you a good continuation. Wishing you the best of luck for all your blogging efforts. <a href="http://www.digitalproductsavenue.com">ebooks with resale rights</a>

Reply
Share

Massive data center failure? What is that really mean?
<a href="http://www.ohio-home-security.com/home-security-systems-columbus-ohio.ht... security systems columbus</a>

Reply
Share

with this statement of yours "Really disappointed with Rackspace. We pay a premium for their service, not because we expect perfection, but because we want 'the best' when something does go wrong."

- yes I am also disappointed with that.. I am more disappointed and yes you know the reason why.
<a href="http://www.incredipuppy.com/happycustomers">puppies for sale in miami</a>
<a href="http://www.incredipuppy.com/aboutus">pomeranian puppies for sale in texas</a>

Reply
Share

maybe the causes of this failure was because of the wrong input in the software or something.
<a href="http://www.johnsoncomfort.com/airconditioning-greenwood-indiana/">repair air conditioning greenwood indiana</a>

Reply
Share

When you're confident, you can achieve your goals. Pretty much interesting indeed. thanks!

Reply
Share

they should have back up in case problems like this will occur again. <a href="http://www.coppervalleydevelopmentgroup.com">custom home builder paradise valley az</a> | <a href="http://www.youtube.com/watch?v=pxjB86jVEcc">custom home builder scottsdale az</a>

Reply
Share

Thanks a lot for the information. It was all helpful for me.
<a href="http://www.altuslift.com/">Forklift</a>

Reply
Share

Post New comment or question

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.

NZ Market Snapshot

Forex

Sym Price Change
USD 0.7786 -0.0046 -0.59%
AUD 0.8852 -0.0021 -0.24%
EUR 0.6215 0.0001 0.02%
GBP 0.4868 -0.0029 -0.59%
HKD 6.0371 -0.0375 -0.62%
JPY 87.2850 1.7180 2.01%

Commods

Commodity Price Change Time
Gold Index 1198.1 -13.300 2014-10-30T00:
Oil Brent 86.2 1.180 2014-10-30T00:
Oil Nymex 81.1 -1.180 2014-10-30T00:
Silver Index 16.4 -0.844 2014-10-30T00:

Indices

Symbol Open High Last %
NZX 50 5370.2 5405.3 5396.7 -0.16%
NASDAQ 4639.4 4641.5 4566.1 1.28%
DAX 9283.4 9339.3 9114.8 2.33%
DJI 17208.8 17381.4 17195.4 1.07%
FTSE 6463.6 6553.4 6463.6 1.28%
HKSE 23913.7 24046.4 23702.0 1.25%
NI225 15817.1 16533.9 15658.2 4.83%