Data center outage causes major flight delays

A system error took down the National Airspace Data Interchange Network (NADIN), a multi-billion dollar data center operated by the FAA, for several hours yesterday. The system outage made thousands of airline flight plans unavailable, meaning thousands of planes could not take off in the United States until the issue was fixed.

The problem has been linked to a router that went down after routine scheduled maintenance. The data center did have a back-up in place, but it failed to start after the main router failed.

This is NADIN’s fourth outage since its introduction in 2002. The system has been heavily criticized for its high cost and poor reliability.

Source | Computer World

Lesson learned: pay your host on time

Paying for any service in a timely manner is always a good idea, but as a VPS user on a web hosting forum discovered, not paying your host could mean losing your data.

In this particular case, the individual was 40 days late on his VPS payment. The host terminated the account, but the person had not been making backups and lost all of his site data. The provider was willing to reactivate the account and restore the files, but only if the customer signed up again at a price $30/month higher than before.

It is never a good idea to pay your hosting fee late, and an even worse idea to trust your host to make backups for you.

Prevent data disasters with a RAID array

Do you make regular backups of your data? Even if you do, you will always experience significant downtime in the event of a hard disk failure on your dedicated server.

One solution to this problem is to upgrade from a single drive to a RAID array. Simply put, the technology involves running two hard drives in tandem. One is used by the server, while the other keeps a constant backup of everything. In the event the main drive fails, the backup takes over.

After experiencing two hard drive failures on my dedicated server, I paid my provider around £100 to have a second hard drive installed for RAID. While a bit pricey up front, the cost of downtime for me was much greater. I not only lost revenue when my sites were down, but also had to spend a significant chunk of time reconfiguring my server and uploading my data. A RAID array was a no-brainer.

Flood ravages Vodafone data center

This isn’t the first data center flood covered here at Internetblog, but most certainly the worst. In the video above, you can watch as several feet of water rush into Vodafone’s Istanbul data center and destroy thousands of pounds worth of server hardware and data.

Immediately after the flood, Vodafone subscribers experienced technical difficulties that lasted 24-48 hours, but no outages were reported. Large tech companies almost always have a system in place where if one data center goes out, the computing load is automatically rerouted to another facility. Needless to say, hopefully the company kept good backups.

Is phone support worth the extra cost?

In an attempt to provide better service, some hosts have returned to the old model of providing phone support as a substitute for the more common online ticket system. It’s great to see technical assistance being provided the old-fashioned way again, but this luxury usually comes at an added cost.

Most hosts don’t offer phone support simply because it’s too expensive. Those that do charge extra for it. This may be a per-call or per-minute basis, a monthly fee, or, in the case of an upper-tier host like mine, implicitly charged to all customers.

You probably won’t need to use phone support most of the time, but it can be useful in a number of situations:

1. Your problem is especially complicated or you want to make sure it is taken care of correctly the first time.
2. Online support has been unable to solve your inquiry.
3. You are technically challenged. We’re not all the next Bill Gates.
4. You don’t have access to a computer at the moment.

There are many benefits to talking to a person live in real-time. If your host offers phone support, why not take advantage of the extra service?

West Africa struggles after cable cut

Yesterday, an undersea SAT-3 cable was cut between the Iberian peninsula and West Africa. It is apparently the only line connecting West Africa to the digital world, and the cut caused connectivity problems in Benin, Togo, Niger, and Nigeria. Nigeria’s banking sector, government and mobile phone networks all suffered from bandwidth outage.

“SAT-3 is currently the only fibre optic cable serving West Africa,” explained Ladi Okuneye, chief marketing officer of Suburban Telecom, which provides the majority of Nigeria’s bandwidth.
“So all West African countries have to use it.”

The fibre optic cable is 15,000km (9,3000miles) long, connecting eight West African countries along is route to South Africa. 70% of Nigeria’s bandwidth is routed through Benin, causing it to suffer greatly from the cut. The company responsible for the network, Suburban Telecom, is sending a ship from South Africa to investigate. According to Okuneye, it could be two weeks before the ship arrives.

Source: BBC News
Photo: Flickr

Iran Crisis Delays Twitter Data Center Upgrade

With the Iranian government blocking most contact with the outside world in the midst of the election crisis, Twitter has become an important line of communication for activists and protesters.

A network upgrade at Twitter’s data center was originally scheduled to take place Monday evening, but now the company has decided to postpone in order to keep the site active and available to Iranians.

Twitter’s data center has had to deal with an unprecedented load because of a traffic increase over the last several months. The upgrade is only expected to take half an hour.

There is speculation that it was not Twitter that made the decision, but pressure from the U.S. State Department. Twitter’s cofounder Biz Stone denied this:

The State Dept does not have access to our decision making process. When we worked with our network provider to reschedule the planned maintenance, we did so because events in Iran were tied directly to the growing significance of Twitter as an important communication and information network. We decided to move the date. It made sense fo [sic] Twitter and [our host] to keep sercie [sic] active during this highly visible global event.

Source: Data Center Dynamics

Data Center's Worst Nightmare Comes True

I hope whoever owns this data center (likely a telecom provider) has good flood insurance. Considering the cost of equipment, time it takes to set up and configure several dozen servers, and most importantly, all the data these machines store, wouldn’t you think the owner of all this stuff would have enough sense to put it somewhere less prone to flooding?

A data center is easy to keep safe if built properly. Believe it or not, web hosts in New Orleans were able to stay in operation after Hurricane Katrina flooded most of the city in 2005. Placing servers above ground goes a long ways towards keeping them dry. I guess some people have to learn things the hard way.

Lightning strike brings outages for Amazon cloud

Amazon.com has released a statement saying that the recent outages experienced on some of their Web services were due to lightning strikes in the United States. On 6:30pm Pacific Daylight Time, a lightning strike caused some of the servers to lose power. This led to disruption with their EC2 service for a limited number of customers.

Such an outage will once again raise questions about the reliability of cloud computing services. On one hand, such outages could have easily happened at any data center, even one locally owned by a small business. On the other hand, users of the services are still putting their data, sometimes of a sensitive nature, at the mercy of the service provider, something that makes some businesses uneasy.

EC2 is a service providing customers with access to Amazon servers using Xen virtualization, a free and open source virtual machine environment for Linux and other Unix-like operating systems. This is not the first service disruption for Amazon’s offerings. Google’s services were also interrupted recently, leading some critics to step to the debate table for a fresh round of anti-cloud computing arguments. Undoubtedly the debate will rage on, and only time will tell how reliably the services can be.

Source: ZDNet Asia
Photo: Flickr

Keeping a close watch on your server

Nothing can be worse than getting a call from an irate client, screaming at you because their web site won’t load. You scramble out of bed, tripping over the cat, and log onto your computer. It only takes you a few minutes to realize that the whole server is down. None of your 20, 50, 100, or even 200 clients on that server can reach their websites. For that moment, your world stops.

There are many good software programs and services available that could remedy this problem. One such solution is called Nagios, a free and open source network monitoring solution. Nagios monitors Windows and Linux/Unix servers, routers, switches, firewalls, printers, services, and applications. You can set it to notify you via email, pager, or mobile phone, and it has escalation capabilities if you would like to have different support personnel notified depending on the severity.

Nagios is available for download from their website. With it, you will be able to rest a little easier knowing that you will at least know when something is wrong before you get that dreaded phone call or support ticket. Nagios was the winner of the InfoWorld Best Open Source Software award of 2008 and the Linux Journal Reader’s Choice award of 2009. A list of other network management and monitoring software is available at Wikipedia.