Revision as of 12:31, 19 October 2013

Note:The correct title of this article is autobackup.pl. It appears incorrectly here due to technical restrictions.

name=unnamed
author=Phil Brady
webpage=none
short=Selective backup of recordings to a small partition.
long=Maintains a backup subset of recordings, taking into account age and whether they are part of a series.
file=autobackup.pl
category=Maintenance
supports=S21:unset,S22:unset,S23:unset,S231:unset,S24:unset,S241:unset,S25:yes,S251:unset,S252:unset,S26:unset,S27:unset,S28:unset,S29:unset

Author

Phil Brady

Description

Maintains a backup subset of recordings, taking into account age and whether they are part of a series.

Description

autobackup.pl selects a subset of mythtv recordings and maintains a backup of them on a destination partition which is typically smaller than the total size of all recordings and which is on a separate disk. It only copies or removes the files necessary to maintain the subset so minimising disk traffic and if run daily will take just a minute or two to complete.

If the destination partition has sufficient space, the script will effectively perform synchronisation of all recordings.

It allows recovery of service with a 'useful' set of recordings in case of primary disk failure and should be augmented by appropriate database and system backups.

Options

--preview or -p

to inhibit file copy/deletes. This shows what action would be taken with a full run but without changing the filestore.

--loglevel n to set logging levels.

n=0 for only errors

n=1 for calling info,

n=2 for stats,

n=3 for a recordings list

n=4 for protocol diagnostics

Default is 3.

--help or -h Help text

--delete <filename>

deletes file from destination directory. For use with the recording delete system event, though use is optional.

Details

autobackup.pl is intended to be run as a regular job when the system is otherwise idle. It checks:

the destination partition to determine how full it is,

the database to extract information about recordings

the source directories as a consistency check

The destination directory to identify recordings which have alreay been backed up and others which are obsolete and need removing.

It is then in a state to identify series recordings, allocate priorities and determine the action for each file.
Files are selected for backup in date/time order (newest first) but:

recordings marked as non-expirable have a 20 year priority boost

recordings which are recognised as episodes of a series can be bunched together with the latest episode and/or have a number of days added as a boost. The author finds bunching with zero days boost to be satisfactory.

Three or more recordings with the same name and an average interval between them of less than 8 days are deemed to be a series.

Files will be promoted from the head of the list, and files demoted as necessary from the end in order to prevent over-filling the partition.

The script should handle recording groups with more than one source directory, though there has only been limited testing of the code for this.

The script was designed to be run automatically at system closedown and to maintain a permanently fitted destination partition, though it should backup satisfactorily to a mounted external eSATA or USB drive if started manually.

Partition Limits

New recordings will be promoted and old ones demoted whilst observing the partition limit of 90%. Older recordings will only be considered if the partition is below a partition threshold of 85% full and this is only likely happen after a spate of manual deletions. The two limits prevent a recording from repeatedly falling in and out of favour as higher priority recordings are created and removed. Threshold should be a few days worth of recordings below limit for stability. The order in which files are copied over by the scipt is chosen for robustness of the algorithm even after an unexpected closedown during copying.

Deleted recordings

Recordings can optionally be deleted from the destination directory with the RecordingDelete system event mechanism by calling this script with --delete %FILE%. This will remove the file more quickly.

However, such files will be routinely deleted by this script even if the event system is not used. Recordings deleted routinely by the script have the advantage of appearing in the log file, complete with recording name, so giving a fuller picture of the change in the state of files.

Recordings which have been deleted from the database but still remain on the destination partition no longer have their name in the database. In order to give these a name in listings, the script writes a list of today's secure recordings to the file to enable the names to be established next day. The file is /var/lib/mythtv2/recordings/autobackup.list.

Working parameters

See the parameters (%params) at the head of the program defining parameters such as the destination partition, maximum percentage used, source directories, destination directories, default logging level, series action, threshold setting and name of recordings list file.

On the author's system, the script is placed in /usr/bin/autobackup.pl. Logging is directed by autobackup.sh to /var/log/mythtv/autobackup.log
The source directory is /var/lib/mythtv/recordings and the destination /var/lib/mythtv2/recordings
The destination partition is /var/lib/mythtv2; the limit is 90% and the threshold 82%.

Logging

If log files are required then they need to be redirected from STDOUT and STDERR and rotated as necessary by a calling script.

Database backups

The perl script has a sister routine autobackup.sh which takes a database backup at last closedown of the day. The sister routine will trigger a recordings backup if it finds this perl script present and manage logging.

Performance

The time taken by this script is dominated by file copying time. With SATA2 interfaces 6Gb/min is a rule of thumb. If the script is run daily, it will typically run in a minute or two.

Glossary

The listing includes these terms:

del deleted from database by the user but still present on the backup partition.

dem or demoted selected for removal from the backup partition in order to make space for promotion(s).

insecure or ins a file which has not been backed up and does not merit it.

promoted or pro selected for copying to the backup partition.

secure or sec a file which is already securely backed up and still merits that.

ser a recording deemed to be one of a series and given a priority boost.

vip a recording marked in the database as non expirable.

Development Regime and Protocol Versions

The script has been developed and tested in the UK on a 64 bit Mythbuntu 12.04 (Mythtv 0.25) dedicated frontend/backend system with ACPI__Wakeup, DVB-T receivers, a single directory for recordings, and a local time close to or equal to GMT.

The intention is to reboot from the 2Tb and run a database restore should the 500Gb fail, and to edit fstab and run Find_orphans.py should the 2Tb fail..

This system has been developed with version 72 of Myth Protocol and it will need validating carefully against other implementations. See Myth Protocol/Guide and Myth Protocol. The author believes that there should be no problems with protocol versions 67 to 75 (myth 0.25 and 0.26) and the keys are included in the code but commented out. To validate against one of these versions uncomment the key (see %MythProtocolKey) and test thoroughly with --loglevel 4 and then with --preview. There were major changes to the program record at version 67 so earlier versions will need more radical changes.

Example logging

This shows the statistics produced, and a (severely cut) list of recordings. The error has been deliberately introduced. See glossary for meaning of status.