Please go here for the most up-to-date version of this howto. i put this howto into my wiki as it's far easier to maintain and a lot more clearly laid out.

INDEX

0. What's this all about
1. How it works
2. Features
3. Portability/Prerequisites
4. How to customise
5. The result
6. The script
7. Download
8. Run it
9. Restore
10. Changelog

0. What's this all about
i want to backup my system quickly, easily and with minimum effort. that's why i wrote this script. the resulting backup file (.tar.bz2 or .tar.gz) is called a stage4. you're probably a gentoo user and hence know the three stages gentoo uses. this backup will be a stage4, meaning, it will be a full system backup of your current system. I got inspired by this thread. if you want to know how it works, continue reading, if not, there's no reason to read all through this thread - jump to Section "8. Run it".

1. How it works
it's a bash script which uses only "standard" tools, namely "tar" and if you intend to burn the stage4 on a cd you'll probably use "split" too. this means this script works out of the box. it has been modified a lot and i now tried to detect any error/misconfiguration a user might do (wrong $PATH, wrong commands, wrong exclude list, overwriting existing files ...).
after checking everything's all set up the script will provide you with four options:

Code:

Backup script v3.5
==================

What do you want to do? (Use CONTROL-C to abort)

Fast (tar.gz):
(1) Minimal backup
(2) Interactive backup

Best (tar.bz2):
(3) Minimal backup
(4) Interactive backup

Please enter your option:

Minimal backup
this option allows you to quickly build a stage4.tar.bz2/stage4.tar.gz without any further fuss, which means
1. it will exclude (i.e. NOT backup) any files/directories listed in the $custom_list variable
2. result in a minimal stage4.

Interactive backup
this option will ask you for any file/directory listed in the custom_list variable if you want it to backup. this might result in the same stage4.tar.bz2/stage4.tar.gz if you answer all question with "no" or in a considerably bigger stage4.

Parameters
the script only shows error on stdout per default. if you want to backup in verbose mode and watch the files processed by tar call the script like:

Code:

./mkstage4.sh --verbose

or

Code:

./mkstage4.sh -v

if you intend to backup your script on a cd/dvd with a predefined size call the script like:

Code:

./mkstage4.sh --split

or

Code:

./mkstage4.sh -s

don't forget to adjust the $split_options variable, i.e. to set the desired chunk size. default is 685MB.

of course you may combine command line parameters in any way.

2. Features
* works right out of the box.
* checks if all needed tools exist.
* checks that any file/folder listed exists (no spelling errors).
* names backups according to its hostname and the date it was created -> unique, meaningful file name.
* prevents overwriting a backup file created on the same day.
* does an integrity check after the stage4 has been created.
* allows fine-tuning and easy change.
* verbose mode.
* command line option --split for splitting the tarball on the fly.

3. Portability/Prerequisites
Against some misconceptions you may use your stage4 for a box with a different CPU as well. If - and only if - you plan to use your stage4 for another CPU type (not just another box with the same CPU), you must check your /etc/make.conf for the CFLAGS entry:

Code:

CFLAGS="-Os -mcpu=athlon -funroll-loops -pipe"

if your CFLAGS looks like this:

Code:

CFLAGS="-Os -march=athlon -funroll-loops -pipe"

you won't be able to use your stage4 for a box with a different CPU type. "march" breaks compatibility, "mcpu" doesn't. there's an easy workaround: replace "march" with "mcpu" and recompile your hole system:

Code:

emerge -e world

i created a stage4 from my laptop (CFLAGS="-Os -mcpu=pentium4 -funroll-loops -pipe"), extracted it on a usb harddisk and am able to boot my amd64 machine. applications work flawlessly.

4. How to customise
as mentioned above there are several variables to customise. the script itself has a few comments which should get you started. you might want to change the location where the stage4.tar.bz2 is put (currently /mnt/backups/stage4), or you might not like the filename (hostname-stage4-datum.tar.bz2), or you might think that the kernel sources is part of a minimal system backup and hence you won't exclude it (and so on) ...

$default_exclude_list: put any file/directory in here which is never needed nor wanted for a minimal full system backup.
$default_exclude_pattern: exclude patterns for the $default_include_folders/files.
$default_include_files: files/folder which are needed for a minimal working system. don't add folders which should be backed up recursively. folders added here provide solely the folder structure.
$default_include_folders: folders which need to be backed-up recursively for a minimal working system.

$custom_include_list: directories which aren't needed but which may be desirable to be backed-up too interactively (like /home or /usr/src/). of course opinions about such files/directories differ, so suit yourself.
$custom_exclude_list: files/folders which are subfolders of a folder listed in $custom_include_list which should NOT be backed up.
$custom_exclude_pattern: exclude patterns for the $custom_include_list.

5. The result
the result is IMPRESSIVE.
stage4.tar.gz: you'll get a stage4.tar.gz which will take about half the time it does for a bzip2 stage4. for my laptop it takes about 20 minutes to create the stage4, my amd64 with a software raid0 needs 8 minutes. unpacking takes about 2/3 of the time it does for a bzip2 stage4. the stage4.tar.gz will be about 10% bigger than a stage4.tar.bzip2. it's your choice!
stage4.tar.bz2: you'll get a stage4.tar.bz2 (for me it's about 800MB from 3.5GB of data) which will be a fully working backup of your system. on my ibm laptop it takes about 50 minutes to create the bz2 stage4.

i recall all those hours i backup-ed my windows installation. the record was 7CD's for a full installation (inkl. some software you need for the daily use) - not bootable .

# verify that each command we use exists. if one can't be found use $PATH and make a suggestion if possible.
for command in ${command_list[@]}; do
if [ ! -x "`which $command 2>&1`" ]; then
echo -e "\nERROR: $command not found! "
base=`basename $command`
if [ "`which $base 2>&1 | grep "no \`basename $command\` in"`" != "" ]; then
echo -e "ERROR: $base is not in your \$PATH."
fi
exit -1
fi
done

help="\nUsage:\n\nsh `basename $0` [[-v]|[--verbose]] [[-s]|[--split]] \n\nTo run the script NOT in verbose mode comes in handy if you want to see only the errors that occur during the backup.\n"

# patterns which should not be backed up (like iso files).
# example: default_exclude_pattern="*.iso *.divx"
# These pattern count only for files NOT listed in the $custom_include_list.
default_exclude_pattern=""

# folders, which need to be backed up recursively on every backup.
# don't touch it unless you know what you are doing! the reason for this
# variable is that some users add /var to the $default_exclude_list. here
# we ensure that portage's memory is backed up in any case.
default_include_folders="
/var/db"

# add files/folders here which are subfolders of a folder listed in $custom_include_list which should NOT
# be backed up. eg.
#custom_exclude_list="/home/foo/mp3 /home/foo/downloads /home/foo/.*"
custom_exclude_list=""

# Only files/folders within the $custom_include_list are checked against these patterns
# custom_exclude_pattern="*.mp3 *.iso"
custom_exclude_pattern=""

Re-assemble split-ed stage4
If you split your stage4 either with the command line parameter --split/-s or uncommented the split section within the script you need to re-assemble the split chunks if you intend to restore such a stage4:

Code:

cat stage4.tar.gz_* > stage4.tar.gz

or

Code:

cat stage4.tar.bz2_* > stage4.tar.bz2

substitute stage4.tar.gz/stage4.tar.bz2 with your stage4 name.

10. Changelog2005-03-23 fixed split command2005-03-27 added new option for faster backup (gzip)2005-03-28 as of udev i added /dev/console /dev/null to be backed up - gets rid of those "Unable to open initial console" messages2005-03-27 added /dev/console, /dev/null to both final_command variables2005-03-31 fixed issue of included files/folder from excluded parent folder2005-04-15 fixed typ (zip -> gzip) in command_list2005-04-20 added new $exclude_pattern variable thanks to a hint by saskatchewan46. not yet checking/verifying valid entries though2005-04-24 NW RLAS. the stage4 is compressed in-place -> no more temporary additional space is needed. it's faster now.2005-04-28 Fixed command (bzip2 options now create a bzip2 stage4 again). small change in the code.2005-04-30 Fixed Restore commands. thanks to Alpo Nestori2005-05-08 Removed warning about the /boot/boot symlink. no longer necessary2005-05-09 Added "6. Warnning" section. $default_include_list doesn't recursively back up folders2005-05-09 Fixed issue with $default_include_list, i.e. added new variable $custom_include_list2005-05-15 nclude /tmp without its content to preserve folder structure. thanks to Lorijho. Verifying custom_include_list for wrong entries.2005-06-07 Added integrity check of the stage4. Changed default filename to Year-Month-ay for easier listing.2005-06-08 Fixed issue about not preserving all permissions.2005-06-11
* Fixed confusing variable names.
* Removed old debug echo statement.
* Split $default_include_list in $default_include_files and $default_include_folders.
* Added portage's memory (/var/db) to $default_include_folders to prevent any mistakes.
* Added portage's /var/cach/edb to $default_include_folders.
* Included files/folders now take precedence of excluded files/folders.
* Not existing files/folders in any list exits the script as it could be ignored too easily.
* Added ftp link to the script for those having issues copy & pasting it.
* Updated this Howo - explained some variables.2005-06-13 Added $custom_include_list, $custom_exclude_list, $custom_exclude_pattern to provide the abilty of backing up directories interactively while excluding files/folders deeper in the tree.2005-06-14
* Added /var/db and /var/cache/edb to the $default_exclude_list to prevent backing it up twice.
* Removed udev warning sections as there seem to be no futher issues.
* Added verbose mode to the script. Default mode is now non-verbose, i.e. only errors are shown. For verbose mode use "mkstage4.sh --verbose" or "mkstage4.sh -v".
* Hardcoded tar and find. Added routine for suggestion path if tar and/or find could not be found.
* Updated Howto. New Section "2. Features". New Section "7. Run it".2005-06-15 Added section "3. Portability/Prerequesites"2005-06-21 Added new parameter --split and new section "Parameters" in "1. How it works".2005-06-30
* Fixed issue with command line parameters and mutual exclusion. --split and --verbose or -s and -v may now be combined in any order.
* Fixed integrity check for splitted files - it's now possible
* Removed /var/cache/edb as this is only cache and portage does not need it to be backed up.
* Removed /var/log/portage per a default as new users get confused about adjusting the script.
* Added used commands to be checked at the beginning.2005-07-25 Added vmware link to dundas mini howto._________________Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick

Last edited by BlinkEye on Mon Nov 14, 2005 10:59 pm; edited 67 times in total

Well that would overwrite your partition table, too. That's not so good idea if you changed partitions meanwhile or just want to restore it on other hard drive.

As far as I know (so it should be checked), last 2 bytes in the MBR tell BIOS to boot or not to boot from that drive (0xAA55 or something else).
64 bytes before that are partition table: 4 entry with 16 bytes each.
So, boot code (lilo, grub, or whatever) shoud be in the first 446 bytes of MBR.

Maybe we should backup, or at least restore, just boot code and try not to mess partition tables...

Code:

dd if=/dev/hdx of=mbr.img bs=446 count=1

But, anyway, I thing that it should be better to chroot in restored environment, edit fstab & lilo.conf, and run "lilo -v" just to avoid partition location and numbering problems..._________________gentoo user

Last edited by tnt on Sun Mar 27, 2005 9:58 pm; edited 1 time in total

notice: new version of the script - it's now possible to choose between a tar.bzip2 or a tar.gz stage4. i had to unpack a stage4 on a 166Mhz box recently where i noticed it might be desirable not to use bzip2 but gzip._________________Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick

i don't know how this happened, i didn't miss that quote in my script even though i do copy the code from my script. nevertheless i found another minor mistake and hence changed the version to 1.3.1. thanks!_________________Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick

Last edited by BlinkEye on Fri Apr 01, 2005 8:22 pm; edited 1 time in total

glad to hear that. i just noticed that /dev/console and /dev/null don't get backed up this way. doesn't work combined with the exclude parameters. hmm, i'm trying to figure out a solution, but at the moment i don't see one without using the -T parameter (which needs an extra file). anyone has got an idea?_________________Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick

a new version: 1.4.0. i finally took the time to look into the issue that you can't backup files/folders under a folder which is excluded. see info page from tar: it's simply not possible, nomatter how you do it. i added a new variable $default_include_list which purpose's to backup files/folder which are deeper in the tree of an excluded folder (like /var/log/messages or /dev/null). it's somehow slower as i must do it in 3 steps now - and you need temporarly more harddisk memory (the compression of the tar file can't be done in-place anymore, but the final stage4 is of the same size of course). nevertheless the script's now even more easily configurable and you won't get those nasty "Could not open inital root= console" errors with udev anymore. if you want to know the running time use it like:

What happened to creating the filesystems? (Here's where I learned that 2005.0 CD for AMD requires ext3 to be mounted as "mount -t ext3...", some bug in the e2fprogs utilities)

For restoration I ran into a number of snags. First, when restoring the partitions, the sequence

Code:

sfdisk -d /dev/(your_disk) > partitions.save

Code:

sfdisk /dev/(your_disk) < partitions.save

works just great if I try onto a second drive in one computer, but when I bring it to another computer, it doesn't work right. I get errors like

Quote:

Partition 1 does not end on cylinder boundary.

The drive where it worked was an Hitachi 400 GB connected via Silicon Image SATA, but the one where it didn't is a WD 74 GB connected via 3Ware SATA). Both systems are Opterons. I'm not sure that could make a difference. In the first case, I was booted on my live filesystem, on the second I was booted from Stage4 DVD.

Interestingly, restoring the MBR after restoring the partitions eliminates partition funkiness, so for some strange reason sfdisk appears to be doing something wrong when it's run from this other computer from my Stage4 DVD (which is based Gentoo 2005.0 CD for AMD64).

However, the MBR didn't completely restore. First, the bootable flag for /dev/sda1 was not set. Second, GRUB just hangs at boot:

You really have made this a great script. How about adding Dar to the script, so if it's finds an older tar, it will dar it and save space._________________veritas vos liberabit...
Linux User Number: 346805
Wine-Wiki
AMD 64 3500+ | MSI "K8T NEO2-FIR" | mushkin Dual Channel DDR 400

@mauricev: thanks for the link. i didn't know such a thread existed, have been looking for it for quite some time.
@greven: seems i'll have to look into dar, looks like a great tool._________________Easily backup up your system? klick
Get rid of SSH Brute Force Attempts / Script Kiddies klick

Thx for the tip on w3m, this is a nice browser. I think u should keep in mind that having a download of the script makes your job harder, because u have 2 update both scripts every time something changes. Copying with w3m is peanuts. So u might want to considder removing the link.