Linux Step By Steps


Using mirrordir for FAST Linux OS recovery
Written by Scott Henderson on 11-March-03.

So you're boss just told you that your Linux web/mail/whatever server can't EVER go down? She wants 100% uptime? Or... you've just built your first Linux server at work, and you're scared you might make a mistake and blow the thing up? Or just want to make recovery easy on your home laptop, where you're always playing and installing new software?  Well, mirrordir can help!  (although it might be stretching things JUST a TINY bit to say we can achieve 100% uptime  :)

This document describes how to set up a periodic, automated replication of all of your important files, including the OS system files, using the mirrordir program, and how you can boot to this replicated version at will, including when the originals are destroyed or corrupted. 


 
mirrordir
is a powerful yet fairly simple utility for mirroring  data on Unix/Linux directories and disk partitions. In this document, mirrordir is used to mirror the / partition on a Linux computer to another partition, usually on the same disk.  So this configuration is NOT designed to protect against hard disk failure (though that could be done with mirrordir too). It is also, of course, not a complete high-availability solution.  What it IS for, is for recovering from:


Recovery using this method is fast - no backup tapes or time-consuming recovery techniques.  All the work is done beforehand so that in the event of a corrupted or otherwise failed Linux OS, you can instantly have the box back, to the state is was a day or 2 previous.   This technique is not designed as a replacement for regular backups (though there will always be those who will use it that way... )


Introduction

First things first... A warning: mirrordir is a powerful tool. If you make a mistake when you set it up, it can send all your data off to NeverNeverLand. Experiment with a test box first and proceed carefully!

If what you are interested in is protection against hard drive failure, you should really use something like hardware-based RAID5. Of course you COULD perfectly well use mirrordir to protect you against hard drive failures, if you want (or if you can't afford RAID technology). Just a few modifications to the config described here and an extra hard drive would do the trick. mirrordir CAN also be used as a complete backup system, as well as in many other ways. Such uses, however, are not explained in this document.  See the mirrordir man page for more on these other uses.

In the configuration below, we will NOT mirror our partitions in real time, but just twice each week, in the wee hours of Tuesday and Friday AM. You can, of course, choose whatever frequency for this you want. (Note: mirroring too frequently can be a problem - you don't want to mirror a misconfiguration -- you always want your mirror data to be somewhat old)  Then, in the event of a problem, the box can easily and instantly be reverted to the previous mirror image state. No restore from backup is required, and no event logs, spooled mail, etc., are lost in the process. Since it is common to do software installs and updates over the weekend, it is a good idea to have a mirror copy on Fridays, so we can revert the box to how it was Friday if we mess up. Then, in this document, we'll do one more for good measure, half a week away.

GRUB, aka the "GRand Unified Bootloader" is used in this configuration as the boot loader. If you use LILO, or another, modify accordingly.

All of mirrordir's activities will be logged in a file called /var/log/mirrordir.log.  You may want to learn about automating log rotation and including this log file in a rotation scheme.  It is not required, however, and the log is very small.

There are 2 ways to boot to the mirrored partition.  One way is to modify the boot loader menu to have 2 options, one for normal boot, one for booting to the mirror.  The other way is to set up a special boot disk, which when inserted will boot the box to it's mirrored partition.

Setup/Configuration

In addition to the normal / and swap partitions, and whatever other partitions you have, for this configuration you will need a free partition at least as large as the / partition, for what we'll call the "mirror" partition. And you will need either:
   -      /var/log and /var/spool each in their own partitions, or
   -      /var in it's own partition.
We'll leave the variable, constantly changing elements in these areas OUT of the mirroring. We're not trying here to mirror things like log files or temporary print or mail data spooled in their various locations under /var. We're only protecting the OS and other program files with this mirror. If we WERE to mirror contents under /var, we'd have to do it in real time to avoid having our mirror constantly out of date, and there are other, better ways to do that anyway.

In the example below, / is at /dev/hda1, and the "mirror partition" is at /dev/hda4. Translate these instructions for your own situation as appropriate.

If you have other partitions, you will need to tailor this configuration to suit. For example, if you have a separate partition for /home, you will probably not want to mirror it to another partition (since what we're trying to accomplish here is just a way to get the box to boot if we mess up the OS, not a complete backup solution). In that case (i.e. you DON'T want to mirror /home), you would alter the mirrordir commands listed below to add an "-i --exclude" parameter for /home, so /home would be ignored. If you DO want to mirror /home, either you'll have to set up yet another partition for a mirror just for /home, and edit the commands below appropriately, or your mirror partition for / will need to be large enough to hold all the data from both / and /home. In this latter configuration, you will also need to consider that in the event of booting from the mirror, you wouldn't want to mirror the /home data back to the original / partition, unless you had the data from /home properly redirected so it doesn't end up on the / partition. If all that doesn't make sense, either don't mess with other partitions, or read man mirrordir .

The Steps... (assuming /dev/hda1 and /dev/hda4)

(you'll need to be logged in as root to do pretty much all of this)

1-- RECORD YOUR CURRENT PARTITIONS

Write down the partition table on paper and make the mirror partition bootable if it isn't (use fdisk, then p to print to screen, copy it down, then use "a" if needed, to toggle to boot parameter on the mirror partition, then use "w" to write it all to disk)

2-- RECORD YOUR CURRENT MOUNTS

Write down the partition mounts, too (use the mount command)

3-- A MOUNT POINT FOR THE MIRROR

Make sure there is a /mnt/hda4 directory in your filesystem, and mount the mirror partition at /mnt/hda4. If the partition to be used for the mirror is already mounted somewhere in the filesystem, just unmount it first. (again, translate for your partitions, e.g. if your mirror partition is at /dev/ida/c0d0p6, create a /mnt/c0d0p6 directory.)  Jot this down in your notes.

4-- A 2ND MOUNT POINT FOR THE PRIMARY PARTITION

Create a directory /mnt/hda1 (for the primary boot partition to go when we boot to the mirror) and don't do anything with it right now (again, translate - if you have / mounted at /dev/ida/c0d0p2, create a directory  /mnt/c0d0p2)

5-- SET UP THE BOOT METHOD FOR THE MIRROR BOOT - 2 METHODS
     (you may use one or both methods)

  5A-- ON THE HARD DISK (i.e., add an option to the boot loader menu)

(these next steps, in section 5A, are not needed if you will be using the boot floppy disk method, just skip to 5B. Also-- you may be forced to use the floppy disk method if your BIOS is unable to boot from your mirror partition - if you aren't sure whether this is the case, don't worry. The BIOS will tell you about it when you test it all out - see the "TESTING" section, below)

Set up the GRUB bootloader to have a pointer to the mirrored partition:
  > copy /boot/grub/menu.lst to /boot/grub/menu.lst.orig

> in /boot/grub/menu.lst copy the first boot section, (i.e. one that
starts with a "title" line) and put the copy at the bottom of the file.
(I'm assuming here the 1st boot section is your default, if not, modify this)

> in the new, copied section:
- rename the "title" in the copied section, inserting
"boot_from_mirror" in front of the current title
- change the xxx in "root=/dev/xxx" to the mirror partition
(e.g., change it from /dev/hda1 to /dev/hda4)
- change the partition in both kernel and initrd lines to reflect the
mirror partition also *
- change any other references from the original / partition, to the mirror

> above all the "boot sections" (those starting with a "title" line), just
comment out any lines that point to the standard / partition, such as:

keytable (hd0,0)/boot/us.klt
or
altconfigfile (hd0,0)/boot/grub/menu.once
or
splashimage=(hd0,1)/boot/grub/splash.xpm.gz

...by putting a "#" symbol in front of those lines

> save the file
* watch out here - we're using GRUB notation for the partitions now, which is different, so you have things like /dev/hda4 referenced as (hd0,3). See " info grub " for full details. Especially read the section on "Naming convention" in the documentation CAREFULLY - if you reference partitions wrong, this whole thing won't work, and you may not be able to boot your system!


  5B-- FLOPPY DISK BOOT (set up a boot disk to boot to the mirror partition)

  (you can skip section 5B, if you don't plan to use a boot floppy for the mirror)

Here's the procedure with Red Hat, modify accordingly if you're using another Linux distribution:

-Pop in a blank floppy, mount it and type mkbootdisk  
-When that's done, go to the floppy and find the file syslinux.cfg
-Locate the line that points to the location for the root file system, e.g. "root = /dev/hda1"
-Edit this to point to your mirror, e.g. "root = /dev/hda4"
-Edit the file boot.msg and change any references in there to the original / partition (e.g. change /dev/hda1 to /dev/hda4)
-Save and exit.
-Make the floppy read only.
-Label it "MIRROR BOOT - MACHINE_NAME" (where MACHINE_NAME is the computer's host name).
-Either keep it in the drive, but not all the way inserted, so it's ready to go, or keep it somewhere safe, where all admins who might need it can get to it.
-When  you're ready to boot to the mirror, you'll just put in the floppy and boot.  To boot normally, you'll remove the floppy and boot.


6-- BECOME A LITTLE mirrordir  MASTER

Read "man mirrordir" to familiarize yourself with the mirrordir program and syntax we'll be using in the next step.

7-- INITIAL MIRROR REPLICATION

Now do an initial copy from / to the mirror partition, making sure to exclude any partitions you don't want mirrored (which always means the mount point for the mirror and the /var stuff, at least). Example command (all on one line!):

mirrordir -i --exclude /mnt/hda4 -i --exclude /var/spool -i --exclude /var/log / /mnt/hda4

You will get several lines of apparent errors, saying mirrordir can't read symlinks under /proc directories. Ignore all this. When it is all done (and it will take at LEAST several minutes for the ol' disk light to settle down!), you can run

df -h

to verify that the size of the data (under the "USED" column) on your mirror partition is greater than that on the original.  It should be, since all the data will have been copied, and there are also additional files that have been copied to the mirror file system that are "virtual" files at the original location, like those in /proc.  These will not show up in the size report for your original partition, but will in the mirror.  If the amount of data on the mirror partiton is less, you haven't mirrored over something you should have.

8-- CREATE MISSING DIRECTORIES

Next, create directories on the mirror partition for all the mount points you excluded in your initial mirrordir command line. Example -- if you had (Of course, replace "mirror_partition" with the appropriate path):
-i --exclude /var/spool
somewhere in the line, you will discover there is no:
/mnt/mirror_partition/var/spool
directory. Create these missing directories like this:
mkdir /mnt/mirror_partition/var/spool 

9-- SET UP FSTABs

Next, set up the fstab files. We'll maintain different fstab files on the / and mirror partitions (and thus we won't mirror this file across, from here on out):
  > copy /etc/fstab to /etc/fstab.orig 

> make sure /etc/fstab mounts the correct partitions for / and the mirror.

Example:
/dev/hda1 / ext3 noatime 1 1
/dev/hda4 /mnt/hda4 ext3 noatime 1 2
...etc

> if your fstab file uses labels (like some Red Hat versions), like this:
LABEL=/ / ext3 defaults 1 1
replace "LABEL=xxx" with the /dev/ equivalent, like this:
/dev/hda1 / ext3 noatime 1 1

> save, and exit the /etc/fstab file

> now edit the mirror partitions's fstab file* and reverse these mount point
references (in 3 places):
Example:
/dev/hda4 / ext3 noatime 1 1
/dev/hda1 /mnt/hda1 ext3 noatime 1 2
...etc

* the mirror partition's fstab file, in the above example, would be
at /mnt/hda4, so to edit it, I would use something like:

vi /mnt/hda4/etc/fstab
10-- THE SCRIPT FILE "MIRROR1.SH"

OK, then we'll set up a script file (we'll call it "/sbin/mirror1.sh") to run mirrordir, and then a cron job to call the script a couple of times per week.
** Note: there IS a program called "mirror" so avoid using THAT as the name of any script or file!

The script file /sbin/mirror1.sh, should look something like this (the mirrordir command is of course on just one line, or lines to be continued will need to have a \ at their end):
#!/bin/sh

# output to a log file, so we have a record...
exec >> /var/log/mirrordir.log
exec 2>> /var/log/mirrordir.log

# make sure an old instance of mirrordir isn't running...
killall mirrordir >& /dev/null
sleep 2
killall -9 mirrordir >& /dev/null

# determine current mirror partition (i.e. the one NOT currently /)
if [ -d /mnt/hda1/boot ] ; then CURRENTMIRROR=/mnt/hda1
elif [ -d /mnt/hda4/boot ] ; then CURRENTMIRROR=/mnt/hda4
fi

echo Start mirrordir...
# timestamp the log file and do the work...
date
echo "Current MIRROR (target partition, not source) is at"/mnt/$CURRENTMIRROR

# here's the place to stop any running programs, if you want...

mirrordir -i --exclude /mnt -i --exclude /etc/fstab -i --exclude /var/spool \
-i --exclude /var/log -i --exclude /proc / $CURRENTMIRROR
date
echo ...End mirrordir

# here's the place to restart any stopped programs...

echo
# end of mirror1.sh file


** Other notes:
* do an "-i --exclude" on any mounted partitions that should not be mirrored
* you can't use the variable $CURRENTMIRROR just any way you please. Be careful that it follows a / character in a path statement. You can't, for instance, make CURRENTMIRROR=/mnt/hda4. It just won't work (I don't know why).
* don't forget to make the script file executable: chmod g+x /sbin/mirror1.sh

11-- SET UP THE CRON JOB

OK, now for the cron job, use: crontab -e


and add this line:

  21 3 * * Tue,Fri /usr/sbin/mirror1.sh

So this will launch our mirror1.sh script each Tuesday and Friday, at 3:21 AM.  Salt and pepper to taste.

12-- TESTING

Do a normal reboot to make sure nothing is misconfigured (box should "bounce" normally - we haven't really changed anything!). After reboot, verify that you have both / and the mirror partition mounted, at the expected mount points, matching the info you copied down earlier.

Now boot to the mirror, either by rebooting and selecting the appropriate item in the boot loader menu, or by inserting the mirror boot disk, depending on which method you're using.

You should boot back and forth between the / and mirror partitions several times, to make sure you understand the procedure and everything works as planned (just use the instructions in the "In Case of Disaster" section, below). You will sometimes see the message during boot that the file system appears to have been shutdown uncleanly, and you will be given the option to check it. It's optional. Checking it is not required, nothing is wrong with it.  Remember, when you are booted to the mirror partition, your / partition will actually be on a different logical partition on the disk. So / won't be / at all, so to speak. :)

If, when you try to boot to the mirror partition the first time, you get an error message on boot something similar to what I got on my Compaq Proliant 800:

"Selected cylinder exceeds maximum supported by BIOS"

Then you can either update your BIOS, if there is one that supports what you are trying to access (that's up to you to research), or you can take the easy way out, like I did, and just make a boot disk (in which case, you can just undo the steps in 5A, and do 5B instead).

In case of Disaster

...or "How to boot to the mirrored partition, in one easy step"

You will need to do one of 2 things:

- If your system uses a boot floppy disk, insert it and boot the box.

- If you aren't using the boot floppy, just boot the box, and at the boot loader screen, select the option that starts with "Boot_from_mirror_..."

...how to revert to the original partition -Since you are currently running from the "mirror partition", you should mirror it BACK to the original / partition, to update and/or repair the original, if needed. The regular /usr/sbin/mirror1.sh script will do this. Run it.

-then boot normally.

-if you are using a boot floppy disk, remove it from the drive and boot.

Routine Maintenance/Administration

-From time to time, just check that files created in / exist and match those in the mirrored partition. You can for example create a test doc somewhere in the filesystem to verify it appears on the mirror after the next sync.

-You may wish to periodically test booting to the mirror and back.


  --- end documentation ---


I hope this is useful to you.  Feel free to contact me, scottlhenderson@yahoo.com, with comments and suggestions!

Everyone is welcome to link to this document, as well as to copy all or any portion of it for their own use and/or to share with others pretty much as they see fit.  More specifically,
this document is provided under the terms of the OpenContent license. See http://www.opencontent.org/opl.shmtl for a copy of this license.
 The author assumes no liability for the use of this information.  See license for details.