Hurricane Sandy – A lesson in disaster recovery

Barrels of diesel fuel to be carried up 17 stories for data center generator. (Fog Creek)

Hurricane Sandy was much larger and more destructive than anyone could have predicted. Besides the tragic loss of life and families displaced, much of lower Manhattan and large parts of the surrounding areas on Long Island and New Jersey are destroyed, flooded, burned down, and made otherwise unusable.

Definition from Wikipedia:
A disaster recovery plan (DRP) is a documented process or set of procedures to recover and protect a business IT infrastructure in the event of a disaster.[1] Such plan, ordinarily documented in written form, specifies procedures an organization is to follow in the event of a disaster. It is "a comprehensive statement of consistent actions to be taken before, during and after a disaster."[2] The disaster could be natural or man-made

Companies that have developed and tested a disaster recovery plan are able to more quickly recover from a catastrophic event and get back online to serve their customers.

Unfortunately with Sandy many companies did not have a well defined disaster recovery plan or the plan did not prepare for such a devastating storm with mass flooding, power outages, and vast wind damage. The Steadfast NYC Co-location facility is offline due to lack of power and damage to their ATM communications network. The websites Gakwer, Huffington Post, Jezebel, Lifehacker, Datagram, BuzzFeed, MarketWatch, and others were taken offline due to power outages and flood damage. A statement from Datagram:

“Unfortunately, within a couple hours of the storm hitting Manhattan’s shores, the building’s entire basement, which houses the building’s fuel tank pumps and sump pumps, was completely filled with water and a few feet into the lobby,” “Due to electrical systems being underwater the building was forced to shut down to avoid fire and permanent damage.”

The websites and services that remained online or came back up quickly utilized their Disaster Recovery Plan and moved to a secondary location or to a distributed environment such as a cloud service. One example is Buzzfeed who recovered quickly:

“Two key things helped BuzzFeed recover: After Hurricane Irene last year, BuzzFeed commissioned an offsite datacenter that replicates everything in near real-time. More recently, the site started using Akamai to cache content. That means that when Datagram was offline, the site and its pages should have stayed up — and many did.”

BuzzFeed restored its site by moving its data to Amazon Web Services.

Extreme Up-time Activities

Some companies that did not have a secondary site resorted to unusual measures to keep their services online. The online software company Fog Creek currently has its employees carrying 55 Gallon drums of diesel fuel up 17 flights of stairs to feed their generators to remain online.(see above picture) The picture to the right is of stairs leading to their basement that contains the now unusable generator fuel pumps and building power.

On Long Island where many are without power, workers are setting up in local libraries and community centers to access power and Internet.

Important Lessons:

Every business no matter how small should have a disaster recovery plan that, depending on the business, should think about what is needed to resume business if a disaster occurs, including:

  • Identify primary assets for business continuity (part of a business continuity plan)
  • Secure data storage with regularly tested backups
  • Maintain offsite backup and secondary redundancy service location at least 35 – 50 miles away from the primary location
  • Develop staffing plans and resources for relocation or replacement

This is just the tip of the iceberg for developing a Disaster Recovery Plan. There are many more resources that can be found on the Internet.

With some planning and a little effort most any business can withstand a disaster type event with minimal disruption to employees and customers.

Sources and More Information:

http://blogs.wsj.com/cio/2012/10/29/wall-street-firms-swing-contingency-plans-into-place/

http://abcnews.go.com/Technology/hurricane-sandy-takes-york-city-data-center-gawker/story?id=17601425

http://www.huffingtonpost.com/2012/10/30/hurricane-sandy-websites-floods-data-centers_n_2046034.html

http://status.fogcreek.com/

https://support.steadfast.net/index.php?/News/NewsItem/View/397

https://www.networkworld.com/slideshow/71861/

This entry was posted in Safe Computing, Uncategorized. Bookmark the permalink. Both comments and trackbacks are currently closed.