Kerpau!

This particular sysadmin is wrapping up a 70+ hour work week. Ugh! I hope you all had a more relaxing week than I did. Happy SysAdmin day!

I’ve had the opportunity to work with a lot of sysadmins of varying experience levels lately and think I have it all figured out. Inexperienced sysadmins expect things to work. Experienced sysadmins expect things to not work. This fundamental mindset leads to vastly different behavior patterns among the two types. If a sysadmin expects a system to fail, s/he will be more likely to perform regular backups, design redundancy into solutions, read the manuals before deployment, test a solution before deployment, test a solution after deployment and so forth. The inexperienced sysadmin reads the marketing materials, believes every word that comes out of the salesperson’s mouth, doesn’t plan for failure.

So when the system ultimately fails, the experienced sysadmin has an entire array of options to choose from. Like backups, failovers and expertise gained from reading the manuals. The inexperienced sysadmin has to resort to trial and error fixes on a live system with no fundamental understanding of the technology. This usually make the issue worse and leaves them in their last resort scenario - bugging tech support. Not good if your job depends on it.

Oddly enough, I found this piece of advice on The Simple Dollar post on Wealth.

Err on the Side of Simplicity Where Possible

It was mentioned in the context of money but is definitely applicable to system design. It’s really a variant of Occam’s razor and one I’ve used many times when considering options in design, deployment and maintenance of complex systems.

All things being equal, the simplest solution tends to be the best one

I have been seeing requests for notified-Compliance_page, a non-existent page in my Web logs lately. They are coming mostly from Google networks with Mediapartners-Google/2.1 as the user agent. Does anyone know what these are about? If so, drop me a line or leave a comment. There isn’t much information on the Web about this and I would love to know more. Thanks!

I was pretty geeked out about this - Are Sysadmins Really that Bad?. I submitted it about two months ago. It’s the only article I’ve ever submitted and it got accepted! Smakynet suggested I write about it because it gives me “street cred” as a geek.

Tonight, I write this as I update a storage area network and iSCSI drivers on several Windows servers. Before I do any significant maintenance on a system, I reboot it. It sounds odd, and I do this even though it extends the time required for maintenance.

The purpose is to tease out errors before I introduce changes into the system. Occasionally, an error condition reveals itself during this initial reboot. This is a good thing because now I know that the error can’t be attributed to the change(s) I was about to make. Had I made my changes and then rebooted the system, it would have complicated the troubleshooting process because I wouldn’t know if the condition existed before the change, or if the change I made caused the condition. This allows me to resolve any issue(s) before continuing with the maintenance and it simplifies the troubleshooting process when problems exist.

When there is a major system outage it helps to have a methodology for dealing with everyone from customers to management during the crisis. Here are some tips I’ve picked up over the years for managing those around you when these situations arise.

Don’t Panic
As a professional, it is your responsibility to keep your cool and focus on the issues at hand. If you are calm, those around you will respond in kind.

Be Nice
There will be someone standing around with lots of rhetorical questions rattling around in their head. There will also be people who feel that this is the perfect time for you to conduct a Computers 101 Q & A session even though it will prolong the outage by distracting you from working on the issue. Resist the urge to bark orders or become a smart Alec. As a sysadmin, outages are your show time. You are more visible to the organization on these occasions than at any other time. Use it to build credibility and goodwill with others in your organization. But, do what you can to let these people know you need to focus your thoughts on getting the systems back to a functioning state.

Prioritize
There are questions that should be dealt with during the outage and questions that should be dealt with after the outage. Recognize the ones that should be deferred and deal with them later. Generally, assigning blame should be dealt with later. It’s different than determining the cause of an issue or gaining understanding of an error condition. It’s not essential to getting a system back to a revenue-generating state.

Keep Others Informed
Keep the relevant people updated. Designate a person to do this if necessary, but it is important to keep those with a stake in the business informed of the situation.

Manage the Panickers
There are plenty of these to go around in any crisis. Communicate with them in a calm, matter-of-fact tone of voice. Keep them focused on issues and specifics. Panickers have a tendency to think and speak in generalities. Anything you can do to focus them on something specific and under their control will help. If appropriate, give them a task to gather information or test a configuration for you.

Exercise Change Control
A system outage is not the time to run Windows Update on a server “since you are rebooting it anyway.” That is, unless you have reason to believe that it will fix an issue. Every configuration change should have a purpose that relates to the issue. Track the changes and roll them back if they don’t work. By taking notes on the changes you make during an outage, you will be able to easily document the outage for later reference.

Get Help from Those Around You
It is useful to have someone who will deflect phone calls and walk-in visitors while you concentrate on the problem. Preferably, this should be someone who can understand and communicate technical concepts to others. Have them explain the situation to users and/or customers without inflaming the situation or causing hysteria. Keep the help desk informed because they will be deflecting most of the user calls away from you. They will also be the ones who gather additional information on the problems for you and call the users back to implement client-side changes. Have another tech around to bounce ideas off of. Another perspective is highly valuable in these situations. It also helps to keep your judgement in check during a high-stress situation.

Use Tech Support
Sysadmins are a proud, self-reliant breed of individuals. Recognize when it is appropriate to enlist the support of outside parties. It’s not a sign of weakness. It’s a sign of good judgement.

Be Opportunistic
The sysadmin’s power is at its peak during a crisis. Use the situation as an opportunity to sell management on the solution (e.g. support contract, software, training, redundant hardware, clustering) that would have prevented this situation or will prevent it from occurring again. It doesn’t have to be something that costs money. Authorization for downtime to conduct proactive maintenance or changes in procedures can be just as valuable as any product on the market.

These ideas have served me well for years, and I hope they help you. If you have other helpful tips, please share them with a comment.

GFI Email Security Testing Zone
http://www.gfi.com/emailsecuritytest/

FILExt - The File Extension Source
http://filext.com

Slashdot - News for nerds. Stuff that matters.
http://slashdot.org

digg
http://www.digg.com

DNSstuff.com - DNS tools, DNS hosting tests, WHOIS, traceroute, ping, and other network and domain name tools.
http://www.dnsstuff.com

SANS Internet Storm Center
http://isc.sans.org

Smart Network Data Services
https://postmaster.live.com/snds/

AOL Postmaster
http://postmaster.aol.com/

Brian Madden - Your Independent Application Delivery Resource
http://www.brianmadden.com

Google Reader
http://www.google.com/reader

OSNews - Exploring the future of computing
http://www.osnews.com

Windows Sysinternals
http://www.microsoft.com/technet/sysinternals

Microsoft Script Center
http://www.microsoft.com/technet/scriptcenter

Google Groups
http://groups-beta.google.com/

SANS Institute - Information and Computer Security Resources
http://www.sans.org/resources

BOFH
http://www.theregister.com/odds/bofh/

Winscripter
http://www.winscripter.com/

Sometimes it’s just great to have a command-line tool to do the work for you. I needed to create a scheduled antivirus scan of a specific directory on a machine but didn’t have the password to the unlock the config GUI for the antivirus client.

Rather than track down the security administrator and talking him out of the password I looked into the program folder and found vscanwin32.com. It’s Trend Micro’s command-line scanning tool and gave me a quick way to get the job done without breaking open the entire configuration. Here are the command-line options.

vscanwin32 [/|-option] Drive:[path[filename|@script]] [Drive:[path[filename]] ..]

option: /S - Scan all files in specified dir and all subdirs.
option: /C - Clean virus-infected files without any prompting.
option: /D - Delete virus-infected files without any prompting.
option: /B - Scan boot/partition area only.
option: /P - Scan hard disk partition only.
option: /NM - Do not scan memory.
option: /NB - Do not scan boot sector/partition area of disk.
option: /NC - Scan only, do not take any action on virus files.
option: /BK[+|-] - Clean virus infected files backup switch.
option: /P=path - Specifiy virus pattern path.
option: /P=file[;file…] - Specifiy virus pattern file(s).
option: /LAPPEND - Open LOG file with append mode.
option: /LR[=file] - Write standard report to the specified file.
Default name “report.log”.
option: /LD[=file] - Write detected files to the specified file.
Default name “detect.log”.
option: /LU[=file] - Write undetected files to the specified file.
Default name “undetect.log”.
option: /LC[=file] - Write clean results to the specified file.
Default name “clean.log”.
option: /LCF[=file] - Write clean fail results to the specified file.
Default name “cfail.log”.
option: /VER - Display the version information.
option: /I - Display more information about scanning results.
option: /NV - Only display the files without virus infected.
option: /CLEANALL - Scan and Clean the file until no virus found.
option: /HOSPITAL=path - Move the virus files to specified directory.
option: /DISKIMAGE[=drive] - Scan disk images. (with or without restoring)
option: /NZ - Do not scan zip files. (eg. PKZIP, ARJ)
option: /NLITE - Do not scan pklite files. (eg. PKLITE, LZEXE)
option: /NMICE - Do not scan softmice files.
option: /NMACRO - Do not scan macro files. (eg. DOC, XLS)
option: /NJAVA - Do not scan java files.
option: /NSCRIPT - Do not scan script files.
option: /NGENERIC - Do not use generic scan.
option: /ZIPACTION - Enable zip virus action in callback.
option: /ZIPBREAK - Enable zip virus break in callback.
option: /VSCHAR=n - Set Character environment type from n .
(1: Tranditional Chinese; 2:Japanese)
option: /VSZIP=n - Set decompress layer.
option: /ACTIVEACTION=n - Enable Active Action and set the ProjectID.
option: /VSTEMP=path - Set the default temporary path.
option: /VSEXT=file - Set/Add process extensions from file.
option: /VSEXEXT=file - Set/Add exclude process extensions from file.
option: /VIRUSNAME - List all detectable virus name. (n: # to read)
option: /UNICODESCAN - Scanning and cleaning with unicode files.
option: /Q - Disable all output message.
option: /RENAME - Rename virus-infected files without any prompting.

Now where is that security guy…

A MiB is a mebibyte. It is 1,048,576 bytes in size and is an attempt to correct the confusion that the term megabyte has suffered. In college they taught us that a megabyte was 2^20 but vendors have adapted the term to their needs for the sake of marketing and usability. I’ll probably keep using megabyte for most of my communications but I have noticed that MiB is being used more frequently. This Novell Connection Magazine article jogged my memory on the term. There is an online type converter available at Easy Calculation.