Monday, October 1, 2007

Playing Hide & Seek with servers


By Nancy Hand

Server upgrades, you gotta love 'em. It doesn't matter if you're upgrading the hardware, the operating system, or a major application like Domino. If everything goes smoothly, it's just because you haven't identified the problem yet.

"Everything seemed fine. That should have warned me."

An example? I waited two years for new hardware to replace some boxes salvaged from the trash. When the servers finally arrived, someone else installed and configured the operating system before I got them.

With brand new hardware, it seemed a little silly to install the old version of Domino so I talked the boss into (finally) letting me upgrade to Domino 7.

I unpacked the installation files and started on the first of the new servers. I stepped through the configuration process and double-checked everything. The installation process then built the directory structure and wrote in the program files. I stopped before starting up Domino. Instead I copied in the Server.ID and Notes.ini files from the existing server. Then I applied the latest Fix Pack. I repeated the process for each new box.

After getting a "base" installation, I was able to install the server-specific third-party applications like mNotes, IntelliWatch, and Department Calendar. The next step was to get the existing files onto the new machine without using Domino. DoubleTake was installed on both the old and new servers to copy the data. The files varied, but each server required at least Admin4.nsf and Names.nsf. I left behind Log.nsf and BusyTime.nsf, or CluBusy.nsf, because they would rebuild when Domino started.

After all the files were copied, I shut down DoubleTake, renamed the old server, assigned it a new IP address, and rebooted it. Then I started Domino on the new server. I let it run long enough to build new log, events, and busytime databases before shutting it down. Then I gave the new server the old server's name and IP address before rebooting it. When Domino restarted, I watched for errors.

Everything seemed fine. That should have warned me.

It wasn't until after the fourth server was migrated that problems became obvious. Sometimes everything worked fine, other times the Notes client and Web browser would insist the servers couldn't be found. Entering the IP address or fully qualified name didn't help. Then, magically, the server would respond. I checked DNS, DHCP, and WINS. The old, temporary, server names were gone. Everything looked good.

A week later Harry went into the server room and re-booted one of the new servers. Since he was standing in front of the server, he didn't expect the server to refuse his login because it wasn't able to find itself. It seems the NIC team had acquired multiple adresses. Instead of both NICs answering to one MAC address, and one IP, they were responding separately. With the NICs properly configured the connection problems vanished.

As I continue to update the rest of the Domino servers, I'm sure another problem, something I haven't thought of, will find its way into the upgrade process.