Kerpau!

Tonight, I write this as I update a storage area network and iSCSI drivers on several Windows servers. Before I do any significant maintenance on a system, I reboot it. It sounds odd, and I do this even though it extends the time required for maintenance.

The purpose is to tease out errors before I introduce changes into the system. Occasionally, an error condition reveals itself during this initial reboot. This is a good thing because now I know that the error can’t be attributed to the change(s) I was about to make. Had I made my changes and then rebooted the system, it would have complicated the troubleshooting process because I wouldn’t know if the condition existed before the change, or if the change I made caused the condition. This allows me to resolve any issue(s) before continuing with the maintenance and it simplifies the troubleshooting process when problems exist.

Microsoft says, “TAPI is not terminal services aware.” Apparently, this applies to using HyperTerminal to connect a serial port to a SAN, router or any other device through an RDP session. It will redirect the local COM ports to the server without any issues though. I lost thirty minutes of quality time to this issue today. Uninstalling Terminal Services resolved the issue. Thank goodness it was just a lab server. Maybe this post will help some other hard-working IT professional get home on time. See TAPI is Not Terminal Services Aware for details.

There are times that that Windows makes it difficult to do tasks that are trivial to accomplish on Linux. Simple text processing is a perfect example. I recently had the need to find out all the unique host IP addresses that had used SMTP to connect to an Exchange server during a period of time. I use an advanced reporting package on my Exchange servers but it is geared toward business decisions and capacity planning. It works very well but does not analyze data at this level which leaves me to find a solution to this problem on my own.

Fortunately, I have GnuWin32 installed on my workstation and it gives me the use of traditional Linux/Unix utilities such as cut, sort, uniq and grep. Looking at the SMTP logs I see that the fields are separated by a space and the IP address of each connection is listed in the third column. So I copy the log files for the period to my desktop computer where I have GnuWin32 installed. I cd to the directory where the log files are located and run the following command:

cut -f 3 -d ” ” *.log | sort | uniq > results.txt

I modify it slightly to find all of the unique internal hosts that have connected to the SMTP server.

cut -f 3 -d ” ” *.log | sort | uniq | grep “172.16.” > results.txt

GnuWin32 makes it a simple and fun task where Windows has no easy equivilant.

When there is a major system outage it helps to have a methodology for dealing with everyone from customers to management during the crisis. Here are some tips I’ve picked up over the years for managing those around you when these situations arise.

Don’t Panic
As a professional, it is your responsibility to keep your cool and focus on the issues at hand. If you are calm, those around you will respond in kind.

Be Nice
There will be someone standing around with lots of rhetorical questions rattling around in their head. There will also be people who feel that this is the perfect time for you to conduct a Computers 101 Q & A session even though it will prolong the outage by distracting you from working on the issue. Resist the urge to bark orders or become a smart Alec. As a sysadmin, outages are your show time. You are more visible to the organization on these occasions than at any other time. Use it to build credibility and goodwill with others in your organization. But, do what you can to let these people know you need to focus your thoughts on getting the systems back to a functioning state.

Prioritize
There are questions that should be dealt with during the outage and questions that should be dealt with after the outage. Recognize the ones that should be deferred and deal with them later. Generally, assigning blame should be dealt with later. It’s different than determining the cause of an issue or gaining understanding of an error condition. It’s not essential to getting a system back to a revenue-generating state.

Keep Others Informed
Keep the relevant people updated. Designate a person to do this if necessary, but it is important to keep those with a stake in the business informed of the situation.

Manage the Panickers
There are plenty of these to go around in any crisis. Communicate with them in a calm, matter-of-fact tone of voice. Keep them focused on issues and specifics. Panickers have a tendency to think and speak in generalities. Anything you can do to focus them on something specific and under their control will help. If appropriate, give them a task to gather information or test a configuration for you.

Exercise Change Control
A system outage is not the time to run Windows Update on a server “since you are rebooting it anyway.” That is, unless you have reason to believe that it will fix an issue. Every configuration change should have a purpose that relates to the issue. Track the changes and roll them back if they don’t work. By taking notes on the changes you make during an outage, you will be able to easily document the outage for later reference.

Get Help from Those Around You
It is useful to have someone who will deflect phone calls and walk-in visitors while you concentrate on the problem. Preferably, this should be someone who can understand and communicate technical concepts to others. Have them explain the situation to users and/or customers without inflaming the situation or causing hysteria. Keep the help desk informed because they will be deflecting most of the user calls away from you. They will also be the ones who gather additional information on the problems for you and call the users back to implement client-side changes. Have another tech around to bounce ideas off of. Another perspective is highly valuable in these situations. It also helps to keep your judgement in check during a high-stress situation.

Use Tech Support
Sysadmins are a proud, self-reliant breed of individuals. Recognize when it is appropriate to enlist the support of outside parties. It’s not a sign of weakness. It’s a sign of good judgement.

Be Opportunistic
The sysadmin’s power is at its peak during a crisis. Use the situation as an opportunity to sell management on the solution (e.g. support contract, software, training, redundant hardware, clustering) that would have prevented this situation or will prevent it from occurring again. It doesn’t have to be something that costs money. Authorization for downtime to conduct proactive maintenance or changes in procedures can be just as valuable as any product on the market.

These ideas have served me well for years, and I hope they help you. If you have other helpful tips, please share them with a comment.

Having used several different magnification utilities I have never found one that I really like until now. ZoomIt is a utility written by Mark Russinovich and it is my new favorite utility. The user interface is intuitive and easily usable. It magnifies the entire screen, unlike the built-in Windows magnification utility where it splits the screen in half. It works well on multiple monitor configurations and even captures magnified screenshots. It also does my favorite thing on Windows - it runs without administrative rights. Perfect!

GFI Email Security Testing Zone
http://www.gfi.com/emailsecuritytest/

FILExt - The File Extension Source
http://filext.com

Slashdot - News for nerds. Stuff that matters.
http://slashdot.org

digg
http://www.digg.com

DNSstuff.com - DNS tools, DNS hosting tests, WHOIS, traceroute, ping, and other network and domain name tools.
http://www.dnsstuff.com

SANS Internet Storm Center
http://isc.sans.org

Smart Network Data Services
https://postmaster.live.com/snds/

AOL Postmaster
http://postmaster.aol.com/

Brian Madden - Your Independent Application Delivery Resource
http://www.brianmadden.com

Google Reader
http://www.google.com/reader

OSNews - Exploring the future of computing
http://www.osnews.com

Windows Sysinternals
http://www.microsoft.com/technet/sysinternals

Microsoft Script Center
http://www.microsoft.com/technet/scriptcenter

Google Groups
http://groups-beta.google.com/

SANS Institute - Information and Computer Security Resources
http://www.sans.org/resources

BOFH
http://www.theregister.com/odds/bofh/

Winscripter
http://www.winscripter.com/

Using Exchange System Manager (ESM), view the properties of the public folder. Under the Exchange Advanced tab deselect the “Hide from Exchange address lists” option. Apply your changes and open Outlook. Find the distribution list in the address book and view the General tab of its properties. Note: You may need to adjust your rights by making yourself the owner of the distribution list. Modify the member list to include the Exchange public folder and you are done!

Sometimes it’s just great to have a command-line tool to do the work for you. I needed to create a scheduled antivirus scan of a specific directory on a machine but didn’t have the password to the unlock the config GUI for the antivirus client.

Rather than track down the security administrator and talking him out of the password I looked into the program folder and found vscanwin32.com. It’s Trend Micro’s command-line scanning tool and gave me a quick way to get the job done without breaking open the entire configuration. Here are the command-line options.

vscanwin32 [/|-option] Drive:[path[filename|@script]] [Drive:[path[filename]] ..]

option: /S - Scan all files in specified dir and all subdirs.
option: /C - Clean virus-infected files without any prompting.
option: /D - Delete virus-infected files without any prompting.
option: /B - Scan boot/partition area only.
option: /P - Scan hard disk partition only.
option: /NM - Do not scan memory.
option: /NB - Do not scan boot sector/partition area of disk.
option: /NC - Scan only, do not take any action on virus files.
option: /BK[+|-] - Clean virus infected files backup switch.
option: /P=path - Specifiy virus pattern path.
option: /P=file[;file…] - Specifiy virus pattern file(s).
option: /LAPPEND - Open LOG file with append mode.
option: /LR[=file] - Write standard report to the specified file.
Default name “report.log”.
option: /LD[=file] - Write detected files to the specified file.
Default name “detect.log”.
option: /LU[=file] - Write undetected files to the specified file.
Default name “undetect.log”.
option: /LC[=file] - Write clean results to the specified file.
Default name “clean.log”.
option: /LCF[=file] - Write clean fail results to the specified file.
Default name “cfail.log”.
option: /VER - Display the version information.
option: /I - Display more information about scanning results.
option: /NV - Only display the files without virus infected.
option: /CLEANALL - Scan and Clean the file until no virus found.
option: /HOSPITAL=path - Move the virus files to specified directory.
option: /DISKIMAGE[=drive] - Scan disk images. (with or without restoring)
option: /NZ - Do not scan zip files. (eg. PKZIP, ARJ)
option: /NLITE - Do not scan pklite files. (eg. PKLITE, LZEXE)
option: /NMICE - Do not scan softmice files.
option: /NMACRO - Do not scan macro files. (eg. DOC, XLS)
option: /NJAVA - Do not scan java files.
option: /NSCRIPT - Do not scan script files.
option: /NGENERIC - Do not use generic scan.
option: /ZIPACTION - Enable zip virus action in callback.
option: /ZIPBREAK - Enable zip virus break in callback.
option: /VSCHAR=n - Set Character environment type from n .
(1: Tranditional Chinese; 2:Japanese)
option: /VSZIP=n - Set decompress layer.
option: /ACTIVEACTION=n - Enable Active Action and set the ProjectID.
option: /VSTEMP=path - Set the default temporary path.
option: /VSEXT=file - Set/Add process extensions from file.
option: /VSEXEXT=file - Set/Add exclude process extensions from file.
option: /VIRUSNAME - List all detectable virus name. (n: # to read)
option: /UNICODESCAN - Scanning and cleaning with unicode files.
option: /Q - Disable all output message.
option: /RENAME - Rename virus-infected files without any prompting.

Now where is that security guy…

A MiB is a mebibyte. It is 1,048,576 bytes in size and is an attempt to correct the confusion that the term megabyte has suffered. In college they taught us that a megabyte was 2^20 but vendors have adapted the term to their needs for the sake of marketing and usability. I’ll probably keep using megabyte for most of my communications but I have noticed that MiB is being used more frequently. This Novell Connection Magazine article jogged my memory on the term. There is an online type converter available at Easy Calculation.

Which one is correct?

Wikipedia uses system administrator.
System Administrator Appreciation Day uses system administrator.
Microsoft uses systems administrator.
Webopedia uses system administrator but also lists systems administrator as a synonym.
Essential System Administration, Third Edition by Æleen Frisch uses system administrator.
League of Professional System Administrators (LOPSA) uses system administrator.
SAGE uses system administrator.

It looks like Microsoft is the odd one of the bunch. Which one do you use and why? Does your job title match? Share your comments!