For step 3 I'd just use a modified form of the manual backup conversion step from the upgrade hints:

Code:

cd /tar zcvf /myth/backup/savedfiles.tar.gz ./etc ./root

Also, rather than making a seperate file in step 2, just make two backups, and the first one will serve as a fallback if something else goes wrong. The rest look fine. The list of things restored in /etc is purposely pretty darn minimal. A bigger thing might be the stuff in the DB, especially settings.

As for "What would you gain"... I dunno... I'm an upgrading fool. My DB goes back to June 1st, 2004, and predates my current hardware.

There are times when I run a backup and can't stay in the room to listen for the "backup complete" sounder. In those cases, I would like to manually run the check_backup script when I return to the room so I can be sure it went ok.

Is there a new style of check_backup.sh script to deal with the *.gz backup files? (used in R5E50 and R5F1)

The posted versions of check_backup.sh are coded for the *.bz2 backup files.

Upcoming in R5D1, the backup and restore scripts now automatically sanity check the results. The builtin sanity checks are much faster than the original rough version posted here, and can also be run manually. The current versions, which can be found in /usr/local/bin are:

If so, then the current scripts are right on our machines now and they should be compatible with the current gz archives.

The current scripts are compatible with files that are uncompress, compressed with gzip, or compressed with bzip2.

The default compression used is gzip but there is a gzip_to_bzip2.sh script to squeeze the extra 20% out of your backups if you're really tight on space.

Since R5D1 every backup and restore gets checked to make sure that all the files requested which are in the file system end up in the archive on a backup, or all the request files which are in the archive end up in the file system on a restore. The checkbackup and checkrestore scripts just expose this verification mechanism so it can be used separately.

- On a similar note, the backup rollover list is now longer to reduce the risk of daily automated backups destroying the last backup from when the system was in a known good state if you're away for a couple weeks.

- Changed the way server restarts are done to be a bit more robust.

- The mybackup and mythrestore scripts now have blacklists of files and directories that can cause problems which they filter out of the supplemental lists.

- Suppressed some harmless messages from tar that seem to panic people.

- Made the mythrestore script warn about ambiguous cases with multiple backup files using different compression. Unlike some of the other warnings, this _should_ worry people since it's one of the more common ways to get a bad restore.

- Made the sanity checking smarter in several ways. In particular DB checking is now less prone to false alarms and handles both ISAM and InnoDB tables.

The following is the backup wrapper script I use before doing any risky experiments on my KM production box. It assumes that you've got idle.sh, that the script is being run as root, that there is a backup disk mounted on /backup which is big enough to mirror the whole contents of my "/" and "/myth" trees, and that the "/backup/root" directory exists (not to be confused with "/root" BTW). Ideally, I'd also shutdown as many of the system daemons as possible to minimize files getting changed while the copy is happening, but generally they're only writing to directories we don't really care about anyway like /var/log.

I've installed idle.sh, but not enstated it in any other scripts yet. I just use it to manually get a quick check on the system state.
Right now I'm trying to figure out what has gone wrong with my system since commenting out the bind-address line lots of bad things have happened.
I am also trying to restore to a backup from March very unsuccessfully, so I ran checkbackup to see how it looks, the list is huge, but it ends with this:

So, I did a rollback.sh and checkbackup from the terminal which looked a lot more realistic, and then restore system from the gui,edit: I aslo tried mythrestore from the command line and the output was this:

Code:

No new style backup file found! Trying to do an old style restore.Doing any needed file updates...

Which doesn't make sense because R5F27 is the sole creator of these backup files-but I still don't have any cards and the checkbackup looks the same after the restore as it did after the rollback:

Most of your problems seem to stem from the missing savedfiles.tar.gz which explains everything but the difference in the table lists. To get this back into some kind of reasonable state start by removing the mythconverg.sql.gz file and then move the *.1 files to the base name.

Also make sure you're doing this as root. If you're using su to become root make sure you use "su -" so it acts like a login shell. This avoids any potential problems with paths and aliases set for the mythtv (or other) user.

Also make sure you're doing this as root. If you're using su to become root make sure you use "su -" so it acts like a login shell. This avoids any potential problems with paths and aliases set for the mythtv (or other) user.

Thank you, I had not read that piece of advice before. The mythrestore was a little more predictable this time after completing the changes to the backup files you recommended, the last lines look like:

Code:

./var/www/mythweb/.htaccess/bin/tar: ./etc/asound.conf: Not found in archive/bin/tar: ./etc/default/aumix: Not found in archive/bin/tar: ./var/lib/alsa/asound.state: Not found in archive/bin/tar: Error exit delayed from previous errorsCompleted the restore of files.Starting the DB restore, this can take a while...Clearing out the existing skeleton...Recreating the db...Restoring the data (long)...Doing any needed db updates...Completed the DB restore.Sanity checking your restore...

Checking for the existance of the backup tar file...Using file /myth/backup/savedfiles.tar.gzBackup tar file exists. Checking the compression...Compression looks OK. Checking backup tar file contents...Generating a list of the backup contents.../bin/tar: Skipping to next header/bin/tar: Skipping to next header/bin/tar: Skipping to next header/bin/tar: Skipping to next header/bin/tar: ./etc/asound.conf: Not found in archive/bin/tar: ./etc/default/aumix: Not found in archive/bin/tar: ./var/lib/alsa/asound.state: Not found in archive/bin/tar: Error exit delayed from previous errorsGenerating a list of the directory contents.../usr/bin/find: ./etc/asound.conf: No such file or directory/usr/bin/find: ./etc/default/aumix: No such file or directory/usr/bin/find: ./var/lib/alsa/asound.state: No such file or directoryComparing directory versus backup contents...Live and saved file lists match.

Checking for the existance of the DB dump file...Using file /myth/backup/mythconverg.sql.gzDB dump file exists. Checking the compression...Compression looks OK. Checking DB dump contents...Generating a list of tables and record counts in the DB dump...Generating a list of tables and record counts in the live DB...Comparing live versus saved tables...Live and saved table lists match.

It seems to have restored appropriately, I have my input cards back, and the mythweb status actually has stuff in the back in the encoder status and schedule boxes.

This was a fresh installed R5F27 without using older versioned backups to upgrade. I don't know that the backup files being off were necessarily linked to the only real thing I've done to the box in a a good while which was commenting out the bind-address to enable a second frontend on the network.
I suppose it's a little late now, how do the backup files get out of sync like that normally? Is there a shortcut finding the log entries that might help you in the future?

Thank you again. Now to enstate a more regular backup routine using...

Looks like there may be something funky with your tar files. You don't normally see "Skipping to next header" messages unless the file has been damaged somehow. The "not found" messages are the generally ignorable ones. Those three files are optional, however, I don't suppress the warnings because that could hide something nasty.

Follow up: I had my fun (shame on me for not doing a backup more often) cleaning up the recordings and such, optimized the database, rebooted, and then ran mythbackup as su - It appears as the the /bin/tar is still a little unhappy.
the last few lines of the backup:

Checking for the existance of the backup tar file...Using file /myth/backup/savedfiles.tar.gzBackup tar file exists. Checking the compression...Compression looks OK. Checking backup tar file contents...Generating a list of the backup contents.../bin/tar: ./var/lib/alsa/asound.state: Not found in archive/bin/tar: Error exit delayed from previous errorsGenerating a list of the directory contents.../usr/bin/find: ./var/lib/alsa/asound.state: No such file or directoryComparing directory versus backup contents...Live and saved file lists match.

Checking for the existance of the DB dump file...Using file /myth/backup/mythconverg.sql.gzDB dump file exists. Checking the compression...Compression looks OK. Checking DB dump contents...Generating a list of tables and record counts in the DB dump...Generating a list of tables and record counts in the live DB...Comparing live versus saved tables...Live and saved table lists match.

That's a harmless warning. The version of the backup script in the next release eats the "/bin/tar: Error exit delayed from previous errors" stuff that needlessly worries most people but leaves the missing file warnings.

You'll notice that the script says "Backup passes all checks." which is what you should really be paying attention to. We've spent a bunch of time over the last few revisions tweaking this so that the final status is as reliable as possible. There is still one well documented issue (a false alarm caused by directories or files replaced with a symlink) but other than that if it complains at the end you should pay attention and if it doesn't you're golden.

- A single unified and inclusive backup file for all file system configuration. The backup consists of two files, the first contains a full dump of the MythConverg database, and the second includes all the configuration files from the file system.

- Inclusive backup, and Selective restoration of files. This means that the backup tries to gather everything it can in the way of configuration files, even if they're not needed for a default restore, and that the restore script picks and chooses just what it needs. As a result any special configuration changes you needed to make should be available in the backup for you to refer to or restore.

- Multi-level rollover of backup files that keeps up to 20 old backups. This is intended to facilitate safe automatic scheduled backups. You can freely purge the old backups if you are very tight on disk space, but unless you go insane with the supplemental backup list or fill your home directories with junk, the maximum space used is less than one 30 minute SDTV recording.

- Customizable supplemental backup and restore lists. This allows you to automatically include extra directories or files in every backup, or automatically have the restore extract additional files.

- Automatic checking of backup and restore results. Every backup and restore includes automated comparisons to make sure that the database and file system and the contents of the backup files agree.

- A restore blacklist and exclusions for safety in the face of silly user tricks. People do the dumbest stuff, even when they're been repeatedly and explicitly told not to. When warning signs aren't enough, making the fence higher can help convince them that it's easier and safer to go around the the pasture with the bull in it rather than through it.

- Utility scripts to help you manage and automate your backups. These include scripts to test your backup and restore results, "rollback" to older backups, recompress your backup files to save space, optimize your database, and check if you can do a backup without interfering with other activities on the machine.

What is backed up and restored by default:

Please note that these lists are inclusive. That means that if you see the name of a directory like ./home in a list it generally includes that directory and everything it contains, files, links, subdirectories (and all of their contents), ... when backing up or restoring files. The exceptions are anything which is blacklisted for safety reasons.

Also note that it is perfectly normal for the scripts to warn about optional files like ./etc/asound.conf or ./etc/default/aumix which are not found.

The asymmetry here is both intentional and purposeful. Having all of your /etc directory saved and available for reference is good, but blindly restoring all of it would be very bad. The inclusive backup and selective restore approach gives us the best of both worlds, and the extra space used is insignificant compared to even a few minutes of video or an albums worth of music.

Where to find things and what they are:

- The backups and supplemental lists are stored in the /myth/backup directory.

- The database dump file /myth/backup/mythconverg.sql.gz is a gzip compressed file of SQL commands which can be run to recreate the database layout and contents.

- The file system backup file /myth/backup/savedfiles.tar.gz is a gzip compressed tar archive which can be unpacked to restore the saved files either individually or in bulk.

- The old archived dump and backup files have names with a numeric suffix from ".1" to ".19" appended. For example /myth/backup/mythconverg.sql.gz.1 and /myth/backup/savedfiles.tar.gz.1 would be your most recent archived backup. If you do a new backup those would be renamed with a ".2" suffix before your current backup became ".1". If you do a rollback ".1" becomes current, ".2" becomes ".1" and so on. These "roll off" at both ends.

- The two optional list files /myth/backup/backup.list and /myth/backup/restore.list are configuration files which contain the names of any files or directories the user wishes to have backed up or restored in addition to the builtin default lists.

- The scripts themselves are kept in /usr/ocal/bin along with most of the other KnoppMyth custom utilities. They are:

/usr/local/bin/backupcommon - library of common settings and funcions/usr/local/bin/checkbackup - utility to check a backup/usr/local/bin/checkrestore - utility to check a restore/usr/local/bin/file_editor.py - library for file editing/usr/local/bin/gzip_to_bzip2.sh - utility to recompress files from gzip to bzip2 format. Can reduce the size of the compressed files by ~10-20% at the cost of a lot of time and CPU power./usr/local/bin/idle.sh - utility to check for other activity on the system/usr/local/bin/mythbackup - the main backup script/usr/local/bin/mythrestore - the main restore script /usr/local/bin/optimize_db.sh - utility to optimize and repair your database tables. /usr/local/bin/restore_fixups.sh - utility used to make certain automatic updates during the restore/usr/local/bin/rollback.sh - utility to discard the current backup and make the most recent archived backup current.

Using the supplemental backup and restore lists:

The backup and restore scripts allow you to specify additional files or directories to be included in your backup or restore, beyond what is already specified by the builtin lists. This is done using the two optional list files "/myth/backup/backup.list" and "/myth/backup/restore.list" described above.

- These files do not exist unless you create them. If one or both of the files doesn't exist, no additional entries are added to the corresponding list.

- The files should only contain the names of anything you want to add to the builtin lists for backup or restore.

- The list files should be owned by and only writable for root.

- Each of the files is a whitespace separated list containing the names of any additional objects to be backed up or restored when the corresponding script is run. The convention is to list one object per line.

- All the entries in these lists should start with a "./" followed by the full path name. This is a traditional safety mechanism which allows you to unpack a root directory based tarball under another directory without unintended consequences.

- Wildcards are not currently supported.

- Avoid redundant and blacklisted entries. They will only cause confusion and generate strange warning messages. Putting "./etc/fstab" in you backup.list is silly because the default list already includes "./etc" and everything under it. Putting it in your restore.list is silly because it's blacklisted for good reason.

- Whenever you add things to the restore.list you run the risk of blowing away a modern working (if not optimal) version of some file with a stale one. It is far better to error to the side of caution and extract the file later manually. There are directions for doing this safely below.

Given the default backup list described above, about the only things extra I've found a need to add to the backup are some custom scripts in the "/usr/local/bin" directory. As a result my current backup.list file looks like this:

Code:

root@black2:~# cat /myth/backup/backup.list./usr/local/bin

Please note - Because they are already included in the backup by the default list, there is no need to specifically mention any files under the /etc, /root or /home directories.

Being selective means that the set of extras I add to the restore is a bit longer. My current list looks like this:

Again when it comes to adding things to the restore list, it is much better to error to the side of caution. You can always extract files from the backup later, using something like this sequence of commands:

- The backup and restore scripts require and try to enforce an "idle" system state while they're running, but there are any number of ways for people to violate this condition. Please make sure you disable any "auto restart" scripts or remote systems which access the DB or file system to prevent conflicts.

- /root and /home/mythtv should be fairly clean or the backups can get very large. For example, having a Subversion tree under one of these can cause serious backup bloat. This is true for any backup system, but keeping a rollover stack makes it that much more important.

- Be careful about what you put in the supplemental backup and restore lists. Both in the size of the lists and their content. For example while restoring ./etc/hosts is probably quite safe, restoring all of ./etc is a really bad idea. Backing up ./usr/local/bin is perfectly reasonable, but trying to backup ./myth with these scripts would be amazingly, incredibly, spectacularly bad (because the backup files are written to /myth/backup and it would cause a feedback loop). The scripts do their best to prevent horrors like this but anyone sufficiently stubborn and crazy could thwart their efforts.

- The backup uses gzip compression for new files, but will also automatically recognize and handle files which are uncompressed, or which use bzip2 compression. The user must be careful that this doesn't lead to ambiguity. Having more than one of "mythconverg.sql" "mythconverg.sql.bz2", and "mythconverg.sql.gz" means that the script has to guess which one to use, and may get it wrong. The order of preference is ".gz", followed by ".bz2", and finally "". While mythrestore checks for and warns about this, you really don't want to find out about it like that.

- The scripts try to provide progress messages for any steps which can take a long time, but don't panic if something takes a few minutes. On the other hand, unless your hardware is very slow, your database is huge, or you've got an incredible amount of junk in your home directory, anything over 15 minutes should raise an eyebrow.

- The scripts were designed and written with simplicity, robustness, speed, predictability and ease of description as guiding principles, and the design trade-offs follow. The script don't have any command line options to keep usage simple. They're list driven because lists are simple to set up and explain. They use gzip compression because it's much. much faster than bzip2. They back up everything and let restore sort it out because that's both simpler and more robust. The goal was to have a couple more moving pieces than a large rock, and we've only added more complicated behavior (like automated compression selection) when absolutely necessary.

- Always verify the results of your backup! I strongly recommend making critical backups (such as the one for an upgrade) from the command line as root so that there is no questions about whether it succeeded or if there were any errors.

- The scripts aren't magical. For upgrades:

- If you don't do a backup there won't be anything to restore.

- You can't do a successful upgrade without a valid and up to date backup.

- The less time between when you make the backup and when you actually do the upgrade, the better. If the delay is more than a couple hours, you should redo the verification step to make sure that the backup is still current.

- If you don't have the /myth partition mounted before entering the 2nd stage install (because you forgot to update /etc/fstab or reconfigure an LVM volume and mount it), the restore script won't be able to find the backup files.

- If you can't use the "Auto Upgrade" option of the installer, or fake it by creating the appropriate semaphore file (/home/mythtv/.upgrade) after using the manual option, the 2nd stage setup won't run the restore automatically. (BTW - you can still do it manually, but some of the post restore configuration won't get done).

Converting an old style backup (from KM version R5B7 or before):

Please note that all the commands shown here must be run as root. See this wiki page for instructions on how to BecomeRoot.

- Make sure you don't have an old "/myth/backup/savedfiles.tar.bz2" or "/myth/backup/savedfiles.tar.gz" file.

Code:

rm -f /myth/backup/savedfiles.tar*

- Make an old style backup using the old mythbackup script installed on your system. Again I strongly recommend doing this from the command line rather than through the menus. This will help you ensure that there aren't any undetected failures which will cause you grief later.

Code:

mythbackup

- Make a new style tarball. Doing this effectively converts an old style backup to a new style backup. Note that the leading ./ on these three directory names is important. Don't forget it, or your backup/restore won't work properly!

Code:

cd /tar zcvf /myth/backup/savedfiles.tar.gz ./etc ./root ./home

- Check that the backup worked! Get the most recent check_backup.sh script from the MythBackupAndRestoreHowTo page on the wiki. and run it:

Code:

check_backup.sh

- Run the following command to check that you don't have multiple compression formats for a given backup filename which will cause confusion later:

It's OK if the savedfiles.tar and mythconverg.sql compression (or lack thereof) are different, however, the files should both show the same current date. On my system it produces these healthy results:

Who is online

Users browsing this forum: No registered users and 1 guest

You cannot post new topics in this forumYou cannot reply to topics in this forumYou cannot edit your posts in this forumYou cannot delete your posts in this forumYou cannot post attachments in this forum