Kerpau!

When there is a major system outage it helps to have a methodology for dealing with everyone from customers to management during the crisis. Here are some tips I’ve picked up over the years for managing those around you when these situations arise.

Don’t Panic
As a professional, it is your responsibility to keep your cool and focus on the issues at hand. If you are calm, those around you will respond in kind.

Be Nice
There will be someone standing around with lots of rhetorical questions rattling around in their head. There will also be people who feel that this is the perfect time for you to conduct a Computers 101 Q & A session even though it will prolong the outage by distracting you from working on the issue. Resist the urge to bark orders or become a smart Alec. As a sysadmin, outages are your show time. You are more visible to the organization on these occasions than at any other time. Use it to build credibility and goodwill with others in your organization. But, do what you can to let these people know you need to focus your thoughts on getting the systems back to a functioning state.

Prioritize
There are questions that should be dealt with during the outage and questions that should be dealt with after the outage. Recognize the ones that should be deferred and deal with them later. Generally, assigning blame should be dealt with later. It’s different than determining the cause of an issue or gaining understanding of an error condition. It’s not essential to getting a system back to a revenue-generating state.

Keep Others Informed
Keep the relevant people updated. Designate a person to do this if necessary, but it is important to keep those with a stake in the business informed of the situation.

Manage the Panickers
There are plenty of these to go around in any crisis. Communicate with them in a calm, matter-of-fact tone of voice. Keep them focused on issues and specifics. Panickers have a tendency to think and speak in generalities. Anything you can do to focus them on something specific and under their control will help. If appropriate, give them a task to gather information or test a configuration for you.

Exercise Change Control
A system outage is not the time to run Windows Update on a server “since you are rebooting it anyway.” That is, unless you have reason to believe that it will fix an issue. Every configuration change should have a purpose that relates to the issue. Track the changes and roll them back if they don’t work. By taking notes on the changes you make during an outage, you will be able to easily document the outage for later reference.

Get Help from Those Around You
It is useful to have someone who will deflect phone calls and walk-in visitors while you concentrate on the problem. Preferably, this should be someone who can understand and communicate technical concepts to others. Have them explain the situation to users and/or customers without inflaming the situation or causing hysteria. Keep the help desk informed because they will be deflecting most of the user calls away from you. They will also be the ones who gather additional information on the problems for you and call the users back to implement client-side changes. Have another tech around to bounce ideas off of. Another perspective is highly valuable in these situations. It also helps to keep your judgement in check during a high-stress situation.

Use Tech Support
Sysadmins are a proud, self-reliant breed of individuals. Recognize when it is appropriate to enlist the support of outside parties. It’s not a sign of weakness. It’s a sign of good judgement.

Be Opportunistic
The sysadmin’s power is at its peak during a crisis. Use the situation as an opportunity to sell management on the solution (e.g. support contract, software, training, redundant hardware, clustering) that would have prevented this situation or will prevent it from occurring again. It doesn’t have to be something that costs money. Authorization for downtime to conduct proactive maintenance or changes in procedures can be just as valuable as any product on the market.

These ideas have served me well for years, and I hope they help you. If you have other helpful tips, please share them with a comment.