|Letters to a Young Manager|
|How do you prepare for a disaster? Hopefully you will only face a server or systems crash. Most IT disasters are that specific and local. If you've done your homework, you have backup, redundancy and fail-over that you regularly test. But what about something larger?|
At 5:00 PM on October 17, 1989, I was sitting at my desk on the ground floor in an office park in San Mateo, CA. Our data center was on the second floor, in an interior, secure space. We had backup systems and power. At 5:04 the ground started to shake. A loud rumble grew, like a New York City subway rushing through the basement of an old building. I was sitting on a chair with wheels that began rolling back and forth like a ship in a major storm. I held on to my desk to keep from being pitched across the room. I looked out the window and saw aluminum light poles shaking like hickory sticks. The overpass to the San Mateo bridge approach was swaying. Cars in the parking lot were moving up and down on waves in asphalt that is not supposed to move that way. A book fell from my bookcase behind me. And then it was over. I was lucky. The bookcase could easily have fallen on me. The building could have collapsed.
A few minutes later my cell phone rang. It was the data center manager who was on his commute home. He was calling from the Bay Bridge.
"I just saw my life pass before my eyes," he said.
"What happened?, I asked, trying to focus.
"A first we thought a plane hit the bridge; everyone changed lanes. Then in my rear view mirror I saw the top deck fall down. Ten seconds earlier and I would have been under it." he said, clearly shaken.
"What happened to the data center?" he asked.
"Let me go check."
I ran upstairs. The CEO and CFO were still in a doorway holding on.
"Are you OK?" we asked each other at the same time.
In the data center the scene was chaos. The mini-computers had all shifted about two feet. Monitors on the shelves above had fallen. The power was dead.
We recovered quickly over the next few days. Luckily the machines and building were not seriously damaged. It could have been worse, as the news reports showed the areas where roads collapsed and cars and drivers crushed.
We learned then that disasters can have a broader context. We needed to plan for systems recovery and business continuity. You may say that an earthquake is rare. I would agree. But serious fires are more common. So the question you need to think about is the fire scenario that takes out your building. How does your business survive that?
When the ground moves, hang on
1. What larger crises or disasters have you or your colleagues experienced? How was your business impacted?
2. When was the last time you tested for the "fire scenario"? What did you learn from the test?
3. Do you have a business continuity plan? When was it last tested?
|For Further Reading: |
Amanda Ripley: The Unthinkable : Who Survives When Disaster Strikes - And Why, (Paperback); 2009
|October 17, 1989 |
San Francisco, 5:04 pm
Loma Prieta earthquake
I was at my desk in San Mateo on a chair with wheels, rolling on the deck of ship in a major storm. Sounded like a subway in the basement. Bookcase behind me. Overpass and light poles shaking like hickory sticks, Asphalt parking lot rolling like waves