Contents ContentsPrev PrevNext Next

Letters to a Young Manager


The Perfect Storm, #26
LTYM > Crisis



Dear Adam,
***
How would I prepare IT for a disaster? The short answer is "more!"

In 1992 I was managing data center operations for a stock market systems provider on the top floor of a skyscraper in downtown NY, near the East River. I had asked about backup capabilities and was assured we had state-of-the-art fail-over computers, and a portable generator under contract.

Late on a Friday in March 1993, the "Storm of the Century" hit the east coast, with a hurricane, storm surge and high tide [1]. The East River flooded downtown Manhattan. Our building soon had 13 feet of salt water in the basement. Every electric and telecommunications circuit was fried.

Fault tolerant computers are useless if there is no power. And stock market info cannot be transmitted without telecommunications. We had two days to recover before the market opened on Monday morning. The first problem was that we couldn't get the generator (or people!) near the building until the flood water subsided. Then once the basement was pumped out, we needed to rebuild all the circuits. Crews worked around the clock.

Miraculously, by late Sunday, power and communications were restored. Then the unthinkable happened. An electrician disconnected the generator at the wrong time and crashed the systems. We missed the market open. And we lost customers. Our brand was damaged.

The plan to avoid it ever happening again was called "the insurance plan". It involved a roof-top generator and UPS electrical backup systems. It had a $3M price tag. It was a painful lesson.

The point is, the insurance should have been purchased years prior, before any disaster. Isn't that the point of insurance?
***
Truly yours,
Ed
________________________

[1] See https://en.m.wikipedia.org/wiki/1993_Storm_of_the_Century

Takeaways:

1) Back-up is only as good as the weakest link;
2) Insurance is something you buy before the disaster.

Discussion Questions:

1) What is the weakest link in your computer systems? Are there any single points of failure?
2) When is the last time you ran a full Disaster Recovery Plan (DRP) test?
3) Does your organization have a business continuity plan that handles not being able to get near the building?
4) What about the "911" scenario, when you lose the building and the people to run it?
5) When is insurance too much insurance? Does your senior management team agree that the "policy" is sufficient?

For Further Reading:





© Copyright 2005, 2024, E. G. Happ, All Rights Reserved.