Using mirrordir for FAST
Linux OS recovery
Written by Scott Henderson on 11-March-03.
So you're boss just told you that your Linux web/mail/whatever
server can't EVER go down? She wants 100% uptime? Or... you've
just built your first Linux server at work, and you're scared you might
make a mistake and blow the thing up? Or just want to make recovery easy
on your home laptop, where you're always playing and installing new software?
Well, mirrordir can help! (although it might be stretching
things JUST a TINY bit to say we can achieve 100% uptime :)
This document describes how to set up a periodic, automated replication
of all of your important files, including the OS system files, using
the mirrordir program, and how you can boot to this replicated version
at will, including when the originals are destroyed or corrupted.
mirrordir is a powerful yet fairly simple utility for mirroring
data on Unix/Linux directories and disk partitions. In this document,
mirrordir is used to mirror the / partition on a Linux computer to another
partition, usually on the same disk. So this configuration is NOT designed
to protect against hard disk failure (though that could be done with mirrordir
too). It is also, of course, not a complete high-availability solution.
What it IS for, is for recovering from:
- accidental misconfigurations
- failed software upgrades
- corrupted system files
- many other situations
that can cause the OS to be unable to boot
Recovery using this method is fast - no backup tapes
or time-consuming recovery techniques. All the work is done beforehand
so that in the event of a corrupted or otherwise failed Linux OS, you
can instantly have the box back, to the state is was a day or 2 previous.
This technique is not designed as a replacement for regular
backups (though there will always be those who will use it that way...
First things first... A warning: mirrordir is
a powerful tool. If you make a mistake when you set it up, it can
send all your data off to NeverNeverLand. Experiment with a test box
first and proceed carefully!
If what you are interested in is protection against hard drive
failure, you should really use something like hardware-based RAID5.
Of course you COULD perfectly well use mirrordir to protect you against
hard drive failures, if you want (or if you can't afford RAID technology).
Just a few modifications to the config described here and an extra hard
drive would do the trick. mirrordir CAN also be used as a complete
backup system, as well as in many other ways. Such uses, however, are
not explained in this document. See the mirrordir man page for
more on these other uses.
In the configuration below, we will NOT mirror our partitions
in real time, but just twice each week, in the wee hours of Tuesday
and Friday AM. You can, of course, choose whatever frequency for this
you want. (Note: mirroring too frequently can be a problem - you don't
want to mirror a misconfiguration -- you always want your mirror data to
be somewhat old) Then, in the event of a problem, the box can easily
and instantly be reverted to the previous mirror image state. No restore
from backup is required, and no event logs, spooled mail, etc., are lost
in the process. Since it is common to do software installs and updates
over the weekend, it is a good idea to have a mirror copy on Fridays,
so we can revert the box to how it was Friday if we mess up. Then, in
this document, we'll do one more for good measure, half a week away.
GRUB, aka the "GRand Unified Bootloader" is used in this configuration
as the boot loader. If you use LILO, or another, modify accordingly.
All of mirrordir's activities will be logged in a file called
/var/log/mirrordir.log. You may want to learn about automating
log rotation and including this log file in a rotation scheme. It
is not required, however, and the log is very small.
There are 2 ways to boot to the mirrored partition. One way
is to modify the boot loader menu to have 2 options, one for normal boot,
one for booting to the mirror. The other way is to set up a special
boot disk, which when inserted will boot the box to it's mirrored partition.
In addition to the normal / and swap partitions, and whatever
other partitions you have, for this configuration you will need a free
partition at least as large as the / partition, for what we'll call
the "mirror" partition. And you will need either:
- /var/log and /var/spool
each in their own partitions, or
- /var in it's own partition.
We'll leave the variable, constantly changing elements in these
areas OUT of the mirroring. We're not trying here to mirror things
like log files or temporary print or mail data spooled in their various
locations under /var. We're only protecting the OS and other program
files with this mirror. If we WERE to mirror contents under /var, we'd
have to do it in real time to avoid having our mirror constantly out
of date, and there are other, better ways to do that anyway.
In the example below, / is at /dev/hda1, and the "mirror partition"
is at /dev/hda4. Translate these instructions for your own situation
If you have other partitions, you will need to tailor this configuration
to suit. For example, if you have a separate partition for /home, you
will probably not want to mirror it to another partition (since what
we're trying to accomplish here is just a way to get the box to boot if
we mess up the OS, not a complete backup solution). In that case (i.e.
you DON'T want to mirror /home), you would alter the mirrordir commands
listed below to add an "-i --exclude" parameter for /home, so /home
would be ignored. If you DO want to mirror /home, either you'll have
to set up yet another partition for a mirror just for /home, and edit
the commands below appropriately, or your mirror partition for / will
need to be large enough to hold all the data from both / and /home.
In this latter configuration, you will also need to consider that in
the event of booting from the mirror, you wouldn't want to mirror the /home
data back to the original / partition, unless you had the data from /home
properly redirected so it doesn't end up on the / partition. If all
that doesn't make sense, either don't mess with other partitions, or
man mirrordir .
The Steps... (assuming /dev/hda1 and /dev/hda4)
(you'll need to be logged in as root to do pretty much all of
1-- RECORD YOUR CURRENT PARTITIONS
Write down the partition table on paper and make the mirror partition
bootable if it isn't (use
fdisk, then p to print to screen,
copy it down, then use "a" if needed, to toggle to boot parameter on
the mirror partition, then use "w" to write it all to disk)
2-- RECORD YOUR CURRENT MOUNTS
Write down the partition mounts, too (use the
3-- A MOUNT POINT FOR THE MIRROR
Make sure there is a /mnt/hda4 directory in your filesystem, and
mount the mirror partition at /mnt/hda4. If the partition to be used
for the mirror is already mounted somewhere in the filesystem, just unmount
it first. (again, translate for your partitions, e.g. if your mirror partition
is at /dev/ida/c0d0p6, create a /mnt/c0d0p6 directory.) Jot this
down in your notes.
4-- A 2ND MOUNT POINT FOR THE PRIMARY PARTITION
Create a directory /mnt/hda1 (for the primary boot partition to
go when we boot to the mirror) and don't do anything with it right now
(again, translate - if you have / mounted at /dev/ida/c0d0p2, create
a directory /mnt/c0d0p2)
5-- SET UP THE BOOT METHOD FOR THE MIRROR BOOT - 2 METHODS
(you may use one or both methods)
5A-- ON THE HARD DISK (i.e., add an option to the boot loader
(these next steps, in section 5A, are not needed if you will be
using the boot floppy disk method, just skip to 5B. Also-- you may be forced
to use the floppy disk method if your BIOS is unable to boot from your
mirror partition - if you aren't sure whether this is the case, don't worry.
The BIOS will tell you about it when you test it all out - see the "TESTING"
Set up the GRUB bootloader to have a pointer to the mirrored partition:
> copy /boot/grub/menu.lst to /boot/grub/menu.lst.orig
* watch out here - we're using GRUB notation for the partitions
now, which is different, so you have things like /dev/hda4 referenced
as (hd0,3). See "
> in /boot/grub/menu.lst copy the first boot section, (i.e. one that
starts with a "title" line) and put the copy at the bottom of the file.
(I'm assuming here the 1st boot section is your default, if not, modify this)
> in the new, copied section:
- rename the "title" in the copied section, inserting
"boot_from_mirror" in front of the current title
- change the xxx in "root=/dev/xxx" to the mirror partition
(e.g., change it from /dev/hda1 to /dev/hda4)
- change the partition in both kernel and initrd lines to reflect the
mirror partition also *
- change any other references from the original / partition, to the mirror
> above all the "boot sections" (those starting with a "title" line), just
comment out any lines that point to the standard / partition, such as:
...by putting a "#" symbol in front of those lines
> save the file
info grub " for full details. Especially
read the section on "Naming convention" in the documentation CAREFULLY
- if you reference partitions wrong, this whole thing won't work, and
you may not be able to boot your system!
5B-- FLOPPY DISK BOOT (set up a boot disk to boot to the
(you can skip section 5B, if you don't plan to use a boot
floppy for the mirror)
Here's the procedure with Red Hat, modify accordingly if you're
using another Linux distribution:
-Pop in a blank floppy, mount it and type
-When that's done, go to the floppy and find the file syslinux.cfg
-Locate the line that points to the location for the root file
system, e.g. "root = /dev/hda1"
-Edit this to point to your mirror, e.g. "root = /dev/hda4"
-Edit the file boot.msg and change any references in there to the
original / partition (e.g. change /dev/hda1 to /dev/hda4)
-Save and exit.
-Make the floppy read only.
-Label it "MIRROR BOOT - MACHINE_NAME" (where MACHINE_NAME is the
computer's host name).
-Either keep it in the drive, but not all the way inserted, so
it's ready to go, or keep it somewhere safe, where all admins who might
need it can get to it.
-When you're ready to boot to the mirror, you'll just put
in the floppy and boot. To boot normally, you'll remove the floppy
6-- BECOME A LITTLE mirrordir MASTER
man mirrordir" to familiarize yourself with the
mirrordir program and syntax we'll be using in the next step.
7-- INITIAL MIRROR REPLICATION
Now do an initial copy from / to the mirror partition, making sure
to exclude any partitions you don't want mirrored (which always means
the mount point for the mirror and the /var stuff, at least). Example command
(all on one line!):
mirrordir -i --exclude /mnt/hda4 -i --exclude /var/spool
-i --exclude /var/log / /mnt/hda4
You will get several lines of apparent errors, saying mirrordir
can't read symlinks under /proc directories. Ignore all this. When
it is all done (and it will take at LEAST several minutes for the ol' disk
light to settle down!), you can run
to verify that the size of the data (under the "USED" column) on
your mirror partition is greater than that on the original. It should
be, since all the data will have been copied, and there are also additional
files that have been copied to the mirror file system that are "virtual"
files at the original location, like those in /proc. These will not
show up in the size report for your original partition, but will in the mirror.
If the amount of data on the mirror partiton is less, you haven't
mirrored over something you should have.
8-- CREATE MISSING DIRECTORIES
Next, create directories on the mirror partition for all the mount
points you excluded in your initial mirrordir command line. Example
-- if you had (Of course, replace "mirror_partition" with the appropriate
-i --exclude /var/spool
somewhere in the line, you will discover there is no:
directory. Create these missing directories like this:
9-- SET UP FSTABs
Next, set up the fstab files. We'll maintain different fstab files
on the / and mirror partitions (and thus we won't mirror this file
across, from here on out):
> copy /etc/fstab to /etc/fstab.orig
10-- THE SCRIPT FILE "MIRROR1.SH"
> make sure /etc/fstab mounts the correct partitions for / and the mirror.
/dev/hda1 / ext3 noatime 1 1
/dev/hda4 /mnt/hda4 ext3 noatime 1 2
> if your fstab file uses labels (like some Red Hat versions), like this:
LABEL=/ / ext3 defaults 1 1
replace "LABEL=xxx" with the /dev/ equivalent, like this:
/dev/hda1 / ext3 noatime 1 1
> save, and exit the /etc/fstab file
> now edit the mirror partitions's fstab file* and reverse these mount point
references (in 3 places):
/dev/hda4 / ext3 noatime 1 1
/dev/hda1 /mnt/hda1 ext3 noatime 1 2
* the mirror partition's fstab file, in the above example, would be
at /mnt/hda4, so to edit it, I would use something like:
OK, then we'll set up a script file (we'll call it "/sbin/mirror1.sh")
to run mirrordir, and then a cron job to call the script a couple
of times per week.
** Note: there IS a program called "mirror" so avoid using THAT
as the name of any script or file!
The script file /sbin/mirror1.sh, should look something like
this (the mirrordir command is of course on just one line, or lines
to be continued will need to have a \ at their end):
# output to a log file, so we have a record...
exec >> /var/log/mirrordir.log
exec 2>> /var/log/mirrordir.log
# make sure an old instance of mirrordir isn't running...
killall mirrordir >& /dev/null
killall -9 mirrordir >& /dev/null
# determine current mirror partition (i.e. the one NOT currently /)
if [ -d /mnt/hda1/boot ] ; then CURRENTMIRROR=/mnt/hda1
elif [ -d /mnt/hda4/boot ] ; then CURRENTMIRROR=/mnt/hda4
echo Start mirrordir...
# timestamp the log file and do the work...
echo "Current MIRROR (target partition, not source) is at"/mnt/$CURRENTMIRROR
# here's the place to stop any running programs, if you want...
mirrordir -i --exclude /mnt -i --exclude /etc/fstab -i --exclude /var/spool \
-i --exclude /var/log -i --exclude /proc / $CURRENTMIRROR
echo ...End mirrordir
# here's the place to restart any stopped programs...
# end of mirror1.sh file
** Other notes:
* do an "-i --exclude" on any mounted partitions that should
not be mirrored
* you can't use the variable $CURRENTMIRROR just any way you
please. Be careful that it follows a / character in a path statement.
You can't, for instance, make CURRENTMIRROR=/mnt/hda4. It just won't
work (I don't know why).
* don't forget to make the script file executable:
11-- SET UP THE CRON JOB
OK, now for the cron job, use:
and add this line:
21 3 * * Tue,Fri /usr/sbin/mirror1.sh
So this will launch our mirror1.sh script each Tuesday and Friday,
at 3:21 AM. Salt and pepper to taste.
Do a normal reboot to make sure nothing is misconfigured (box should
"bounce" normally - we haven't really changed anything!). After reboot,
verify that you have both / and the mirror partition mounted, at the
expected mount points, matching the info you copied down earlier.
Now boot to the mirror, either by rebooting and selecting the appropriate
item in the boot loader menu, or by inserting the mirror boot disk,
depending on which method you're using.
You should boot back and forth between the / and mirror
partitions several times, to make sure you understand the procedure
and everything works as planned (just use the instructions in the "In
Case of Disaster" section, below). You will sometimes see the message
during boot that the file system appears to have been shutdown uncleanly,
and you will be given the option to check it. It's optional. Checking
it is not required, nothing is wrong with it. Remember, when you are
booted to the mirror partition, your / partition will actually be on a
different logical partition on the disk. So / won't be / at all, so
to speak. :)
If, when you try to boot to the mirror partition the first time,
you get an error message on boot something similar to what I got on my
Compaq Proliant 800:
"Selected cylinder exceeds maximum supported by BIOS"
Then you can either update your BIOS, if there is one that supports
what you are trying to access (that's up to you to research), or you can
take the easy way out, like I did, and just make a boot disk (in which
case, you can just undo the steps in 5A, and do 5B instead).
In case of Disaster
...or "How to boot to the mirrored partition, in one easy step"
You will need to do one of 2 things:
- If your system uses a boot floppy disk, insert it and boot the
- If you aren't using the boot floppy, just boot the box, and at
the boot loader screen, select the option that starts with "Boot_from_mirror_..."
...how to revert to the original partition
-Since you are currently running from the "mirror partition",
you should mirror it BACK to the original / partition, to update and/or
repair the original, if needed. The regular /usr/sbin/mirror1.sh script
will do this. Run it.
-then boot normally.
-if you are using a boot floppy disk, remove it from the drive
-From time to time, just check that files created in / exist
and match those in the mirrored partition. You can for example create
a test doc somewhere in the filesystem to verify it appears on the mirror
after the next sync.
-You may wish to periodically test booting to the mirror and
--- end documentation ---
I hope this is useful to you. Feel free to contact me, firstname.lastname@example.org, with comments and suggestions!
Everyone is welcome to link to this document, as well as to copy all or
any portion of it for their own use and/or to share with others pretty much
as they see fit. More specifically, this document is provided
under the terms of the OpenContent license. See http://www.opencontent.org/opl.shmtl
for a copy of this license. The author assumes no liability for
the use of this information. See license for details.