Consolidating information technology

I spent about 20-hours these past 2-weeks consolidating services here at RAH-CoCo’s from two old servers (dual PIII and P4 XEON) to a single server using an AMD Opteron.  Those services include:

  • DNS serving for two primary domains: hurst-ri.us and rah-cocos.com
  • Email using sendmail for those domains, with MailScanner, ClamAV, and SpamAssassin
  • Web sites using Apache for those domains, with a MySQL database for WordPress
  • Multi-user dungeon crawl server using RPGD
  • Point-of-Sale database application using Caché
  • File and print sharing using SAMBA for the store’s PCs
  • Many cron scripts for scheduled tasks, automation, and backups

The old servers were SCSI-based and one of them even had RAMBUS memory in it. Needless to say, its components have served its purpose for the past 9 (PIII) and 7 (P4) years.  The newer replacement has a 64-bit AMD Opteron processor, which runs very efficiently. I was able to re-use the PIII case, because the ASUS mobo and processor used an ATX connector and its 350W power supply is ample. The operating system is, of course, Linux, which also provides a nice cpuspeed service that allows this CPU to run at half speed (1gHz) during low workload periods.  Its sensors report overall lower temperature, fan RPMs, and thus making for cooler, quieter, and less costly power consumption. When workload demand escalates, it throttles up to 2gHz, with imperceptible latency.

To help avoid in single point-of-failures, I decided to implement a software RAID mirror between two 120GB drives (one on EIDE and the other SATA-II) while pushing scheduled backup for archiving on a 160GB drive. Again, Linux provides excellent software RAID tools to create and manage RAID disk sets, and again, there is no perceived loss in performance. The act of creating the mirror and sync’ing the partitions is a well-documented process … so its mileage does NOT vary between its implementations. The process and its results are both highly predictable.

The boot-up process, however, is not nearly as straight. There is the BIOS to test for managing around a future disk failure. There is the boot loader (called GRUB) that needs the same kind of attention. And then there is the Linux boot process that transitions from the kernel load and its execution of an initial ram disk, that upon completion, then transfers into a traditional UNIX init run level. This process, too, is also well-documented. But just as it is Linux’s greatest strength, since the scope must allow for a high degree of flexibility — given the innumerable permutations of platforms and components — it is complex. So, working a variety of tools and methods are required to shape for proper implementation.

The crux of this implementation was the making of the new initial ram disk.  Fedora provides a script, mkinitrd, to automate its construction.  Since the server image was originally installed as a single drive, but is now expected to boot into mirrored disks, it is required to make a new kernel ram disk image with appropriate startup instructions.  Sadly, that took a few hours to consider, attempt, and watch fail over-and-over.  It wasn’t until I leveraged the power of Open Source (vi /sbin/mkinitrd) combined with my own skill — and a lot of stubborness I call perserverance — did I finally trace down the problem to an undocumented command-line switch called –fstab.  Fedora’s mkinitrd is so “clever” that it references the disk mounting instructions maintained in a configuration file, /etc/fstab, to construct the pre-init script.  But, if you are making a “new” /etc/fstab (on the mirrored disk, but it is temporarily mounted under the rescue filesystem, i.e., /mnt/sysimage), then you need to point mkinitrd at THAT fstab, so it can get a clue that it needs to make the software RAID devices, such as /dev/md0.  DOH!!

Well, among a great many of other things, this web-site is now hosted on that “new” server. Bringing WordPress and MySQL content over was a fairly straightforward process, even though it was coming from 32-bit Fedora 8 to 64-bit Fedora 9. All other services copied over using similar methods of inspection and integration.  I highly recommend the use of rsync for its file compare & transfer capabilities and the power of a visual difference and editing utility in meld.  Both tools made for great efficiency for attaining a high degree of inspection and confidence that ‘The Right Thing’ was happening during every step of the migration.

What do I do with the old servers?  I get to examine the condition of those (expensive) server boards, and piece out any leftover cards and drives that may serve someday as interim replacements to failed components.  The rest will probably be offered up on ebay — my junk is somebody else’s treasure, for certain.  This experience for me instills further amazement by how far computer technology has evolved over my past 26-years in the field.