Autobackup.pl

Note:The correct title of this article is autobackup.pl. It appears incorrectly here due to technical restrictions.

name=unnamed
author=Phil Brady
webpage=none
short=Selective backup of recordings to a small partition.
long=Maintains a subset of recordings, taking into account age and whether they are part of a series.
file=autobackup.pl
category=Maintenance
supports=S21:unset,S22:unset,S23:unset,S231:unset,S24:unset,S241:unset,S25:yes,S251:unset,S252:unset,S26:unset,S27:unset,S28:unset

Author

Phil Brady

Description

Maintains a subset of recordings, taking into account age and whether they are part of a series.

Supports

THIS PAGE IS UNDER CONSTRUCTION. PLEASE IGNORE IT FOR NOW!

DESCRIPTION:

autobackup.pl selects a useful subset of mythtv recordings and makes a backup of them to a destination partition which is typically smaller than the total size of all recordings and which is on a separate disk. It allows recovery of service with a 'useful' set of recordings in case of primary disk failure. It should be augmented by appropriate database and system backups.

OPTIONS:

--preview or -p

to inhibit file copy/deletes. This shows what action would be taken with a full run but without changing the filestore.

--loglevel n to set logging levels.

n=0 for only errors

n=1 for calling info,

n=2 for stats,

n=3 for a recordings list

n=4 for protocol diagnostics

Default is 3.

--help or -h Help text

--delete <filename>

deletes file from destination directory. For use with the recording delete system event, though use is optional.

DETAILS:

autobackup.pl is intended to be run as a regular job at system closedown when the system is otherwise idle. It checks:

the destination partition to determine how full it is,

the database to extract information about recordings

the source directories as a consistency check

The destination directory to identify recordings which have alreay been backed up and others which are obsolete and need removing.

It is then in a state to identify series recordings, allocate priorities and determine the action for each file.
Files are selected for backup in date/time order (newest first) but:

recordings marked as non-expirable have a 20 year priority boost

recordings which are recognised as episodes of a series can be bunched together with the latest episode and/or have a number of days added as a boost. The author finds bunching with zero days boost to be satisfactory.

Three or more recordings with the same name and an average interval between them of less than 8 days are deemed to be a series.

Files will be promoted from the head of the list, and files demoted as necessary from the end in order to prevent over-filling the partition.

The script should handle recording groups with more than one source directory. The author has made limited tests of this scenario but only uses a single directory.

PARTITION LIMITS

New recordings will be promoted whilst observing the partition limit of 90%. Older recordings will only be considered if the partition is below a partition threshold of 85% full. This is only likely happen after a spate of manual deletions. The two limits prevent a recording from repeatedly falling in and out of favour as higher priority recordings are created and removed. Threshold should be a few days worth of recordings below limit for stability.

DELETED RECORDINGS

Recordings can optionally be deleted from the destination directory with the RecordingDelete system event mechanism by calling this program with --delete %FILE%. This will remove the file more quickly.

However, such files will be routinely deleted by this script even if the event system is not used. Recordings deleted routinely by the script have the advantage of appearing in the log file so giving a fuller picture of the change sin the state of files.

Recordings which have been deleted from the database but still remain on the destination partition no longer have their name in the database. In order to give these a name in listings, the script writes a list of today's recordings to the file to enable the names to be established next day. The file is /var/lib/db_backups/autobackup.list. This directory is used to hold backups and control files from the sister database backup routine autobackup.sh.

WORKING PARAMETERS

See the parameters (%params) at the head of the program defining parameters such as the destination partition, maximum percentage used, source directories, destination directories, default logging level, series action, threshold setting and name of recordings list file.

On the author's system, the script is placed in /usr/bin/autobackup.pl. Logging is directed by autobackup.sh to /var/log/mythtv/autobackup.log
The source directory is /var/lib/mythtv/recordings and the destination /var/lib/mythtv2/recordings
The destination partition is /var/lib/mythtv2; the limit is 90% and the threshold 85%.

LOGGING

If log files are required then they need to be redirected from STDOUT and STDERR and rotated as necessary by a calling script.

DATABASE BACKUPS

The perl script has a sister routine autobackup.sh which takes a database backups. If this script is found then it will trigger it at last closedown of the day. See autobackup.sh.

PERFORMANCE

This script takes negligible time to run, though file copying is a more lengthy operation. With SATA2 interfaces 6Gb/min is a rule of thumb. If the script is run daily, it will typically run in under a minute.

DEVELOPMENT REGIME

The script has been developed and tested in the UK on a 64 bit Mythbuntu 12.04 (Mythtv 0.25) dedicated frontend/backend system with ACPI wakeup, DVB-T receivers, a single directory for recordings, and a local time close to or equal to GMT.

This system has been developed with version 72 of mythtv protocol and it will need validating carefully against other implementations. To do so, you will need to add a new version and its key in %MythProtocolVersion and test thoroughly with --loglevel 4 and then with --preview.

TYPICAL LOGGING

This shows the statistics produced, and a (cut) list of recordings. The error has been deliberately introduced.
The first column of status shows where the file is: sec for secured (already backed up), pro for promote, dem for demote from backed up status and ins for insecure (not backed up). This is followed by VIP (non expirable), ser (series member) or del (deleted).