Revision as of 21:47, 6 October 2010

Storage Groups are lists of directories that are used to hold MythTV recording files. By default, there is only one Storage Group, called "Default".

Note: At least one recording directory has to be added to the Default Storage Group or it will not be possible to watch/record anything with MythTV.

Additional Storage Groups can be created to store specific recordings in their own directories. Storage Groups are edited via the 'Storage Directories' section of mythtv-setup. It may be desirable to add multiple directories (which are mounted on different hard drives) to a Storage Group in order to spread out filesystem I/O onto multiple filesystems.

Multiple Storage Groups can also be created to group recordings together; recording schedules now have an option to specify which Storage Group to use.

MythTV will balance concurrent recordings across the available directories in a Storage Group in order to spread out the file I/O load. MythTV will prefer filesystems that are local to the backend over filesystems that are remote until the local filesystem has 2 concurrent recordings active or other equivalent I/O, then the next recording will go to the remote filesystem. The balancing method is based purely on I/O, MythTV does not try to balance out disk space unless a filesystem is too low on free disk space in which case it will not be used except as a last resort.

Storage Groups are global, but can be overridden on a slave backend by creating a local Storage Group by running mythtv-setup on the slave. If a problem occurs and the slave backend is unable to use the desired Storage Group, it will fail back and try the directories defined in the master's Storage Group.

A special 'LiveTV' Storage Group may also be defined. If a LiveTV Storage Group directory exists, it will be used instead of putting LiveTV recordings in the Default Storage Group, allowing LiveTV recordings to be stored on their own filesystem. This is similar to the old MythTV method which used a RingBuffer for LiveTV.

To remove a storage group from the list, highlight it and press the 'D' key. This does not "delete" the files, it only removes the directory as a storage location. If the storage group you wish to remove is not empty (which is likely the case as MythTV balances the load), you can move these files from one storage group to another with a regular filesystem move. The next time MythTV tries to access these files, it will automatically search all available storage groups to locate them.

Usage information for all Storage Group directories is visible on the mythfrontend status screen as well as the mythbackend status webpage. MythTV is smart enough to determine which directories are on shared filesystems so it should not count free or used space multiple times if more than one directory is on the same filesystem.

Setting up Storage Groups

You should only define Storage Groups on the Master Backend (ceryainly for 99.9% of users).

A Storage Group is simply a mapping between a user-friendly name
("Default", "Movies", "Archive," ...) and a list of directory paths.
This does not say anything about them containing information about the host that contains those directories.

Therefore, when you *define* a Storage Group (which can only be done on
the master backend) and set up a list of directories, that Storage Group
exists on *all* hosts. Note, also, that I did not say that all the
directories in the directory list must exist on all the hosts--only that
the Storage Group exists on all hosts (the /mapping/ between a
user-friendly name and a list of directory paths exists on all hosts).

When you use mythtv-setup on any host other than the master backend to
edit Storage Groups, all you're doing is overriding the directory list
for an already-defined Storage Group--you're saying, "The list I gave on
the master backend is wrong." (So, really, you should just fix the list
on the master backend...)

This is true even in the situation where a slave with it's own storage.

You do not have to run NFS or Samba on my network.

Example 1

Consider the situation where a Master Backend has 4
file systems it's using for recordings and a Slave Backend that has 3 file
systems it's using for recordings.

The Master Backend has no access to the file systems on the Slave Backend and vice versa.

Regardless of this, *only* define Storage Groups on the master
backend. In this example, the Default Storage Group has a directory list:

where a, b, c, d are mount points and actual recordings are put into
subdirectories so that the backend(s) can tell if the file system is
mounted or not (otherwise, if a, b, c, d were put in the directory list, it
would write to the directory whether or not the file system is
mounted--meaning it would fill up the parent file system, which is my
root file system).

As mentioned before, in this example the Master Backend has 4 file systems
and the Slave Backend has only 3 file systems. This isn't a problem.
On Master Backend, a different file system is mounted at a, b, c,
and d. On the Slave Backend, we not (yet) mount a file system at d.
Therefore, the Slave Backend has directories:

Since there's no file system mounted at d, there's no
/srv/mythtv/tv/d/recordings (which is the one the Default Storage Group
uses), so MythTV sees that the directory doesn't exist and doesn't use
it (it simply ignores it) on the Slave Backend.

This means that we have no Storage Groups/Storage Group directory entries
defined for any host other than the Master Backend.

There is no reason to ever touch Storage Groups on any host other than
the master backend.

If you prefer, you can do things slightly differently. Let's say you
don't want to "re-use" directory names on your different backends
because you may find it confusing to have 2 different file systems using
the same directory path. In that case, the Storage Group definition on
the master backend should contain the "union" of directory paths used by
all backends. So, it may have something like:

(where picking directory names is the hard part, but the above gives you
the idea). Again, a, b, c, d are all mount points and recordings is a
directory existing within the file system mounted there. Then, since
there's no /srv/mythtv/tv/master/a/recordings directory on the remote
backend, and no /srv/mythtv/tv/remote/c/recordings directory on the
master backend, MythTV will know to ignore those directories on the
hosts in question.

Example 2

For the situation where a Slave Backend has no storage and you use NFS or Samba,
you may have all your storage on the master
backend or some NAS or whatever. If so, just list all the directories
in the Storage Group definition on the master backend and other hosts
will use the same list of directories. Since all recording is done to
"local" file systems, the remote file systems must be mounted at the
same absolute path on all hosts.

If you feel you must have the TV directories NFS mounted on each MythTV
system, you can, but you need to choose an appropriate Storage Groups
scheduler.

There's no need for any backend to have access to any
file system other than the file systems to which that backend records
TV. All MythTV recordings access (recording, playback,
etc.) is performed through the recording host's mythbackend. The only
exception is when you enable the setting:

Master Backend Override

If enabled, the master backend will stream and delete files if it finds
them in the video directory. Useful if you are using a central storage
location, like a NFS share, and your slave backend isn't running.

This is only meant to be used when you have a single file server
containing all file systems and you want to be able to stream/playback
recordings from remote backends even when the remote backends are shut
down or when *all* of your storage exists on the master backend and
remote backends record to it over the network--so without this setting,
streaming from the recording backend would mean the recording backend
reads a recording over NFS and then sends it across the network to the
frontend. Instead, enabling the override allows the master backend to
read it from local storage and send it across the network to the
frontend (so, only one trip across the network).

Quick Rules

Only set Storage Group directory lists on the master backend. (Never create Storage Group directory list overrides on the remote backends.)

Once properly configured (with no SG overrides on the remote backends), only manage Storage Groups using mythtv-setup on the master backend

Make the master backend's Storage Group directory list contain the union of all directories on all hosts--whether those directories exist on all hosts or not.

Directories in a Storage Group directory list need not exist on any host (those that don't exist on a particular host are simply ignored).

If you don't want a directory used on a particular host make sure that directory doesn't exist on that host.

Never put a mount point into a Storage Group directory list--always put some subdirectory under the mount point so MythTV will know to ignore the (non-existent) directory entry if the file system fails to mount properly.

If you think you need to override a SG directory list on a remote backend or remote frontend, you're making things more difficult than they need to be.

Default Storage Group

The Default Storage Group is special. Users don't define it. It always exists because it's defined in code.

Directory Structure

Note: When specifying directories for Storage Groups, never put a mount point directory name (the root of a mounted file system) in your SG directory list.

If you put a mount point directory name into an SG directory list, MythTV cannot tell when the file system is not mounted (the mount point and the mounted file system directory are identical), so it will gladly write to the parent file system (and possibly fill it up--which is very bad if the parent file system is the system's root file system). Instead, always create one or more subdirectories within your file systems and put a subdirectory path into the SG directory list. Then, when the file system isn't mounted, the directory doesn't exist, so MythTV will ignore it.

Slave Backend Overrides

While it is possible to override a Storage Group in a slave back end, it is not a good idea to do so. Only define your Storage Group once on the master backend and don't override the directory list on the remote backends.

Deleting Overrides

To delete a Storage group Override that has been set up on a Slave backend, go to mythtv-setup, then in Storage Directories.
Highlight the Default Storage Group and then hit your DELETE key.
A pop-up appears that says, "Delete 'Default' Storage Group? (from remote hosts)".
When you say, "Yes, delete group," it will delete the Default Storage Group directory list overrides from your configuration.

The Default Storage Group and the directory list on the master backend will remain.
If you want to edit the Default Storage Group directory list on the Master Backend,
you have to do it on the Master Backend.

Expiration and Free Space settings

The "global" setting "Extra Disk Space," specifies the amount of extra space to keep available on each filesystem, not a total amount of extra space to keep available across all filesystems. Therefore, if the "Extra Disk Space" is set to 3GB, it should be keeping 3GB available on all filesystems (to which MythTV is recording at any particular time).

That parenthetical can actually be very important. If MythTV is not recording to a filesystem, it does not autoexpire programs to make room for the data that some other process is writing to that filesystem. Therefore, until a recording starts and is being written to the filesystem in question, no programs will be expired from that
filesystem.

MythVideo

Videos in MythTV can be stored on the backend and streamed to the frontend without a NFS or Samba mount.

Note: Contrary to recording Storage Group configuration, the video scanner relies on directories being defined on each Backend which houses videos. If you have two backend hosts, each with videos, then you must define the paths to videos on each.

Advantages

Can dynamically add space to your video library without using RAID or LVM. Loss of one drive does not mean loss of the entire library.

Can spread video hosting across many/all backends.

No need to set up network mounts of any kind.

Disadvantages

External Video Players (mplayer, xine, VLC) will not work with videos hosted on an SG.

Setting Up Video/Image Storage Groups

On the backend to host the videos, stop the backend process and run mythtv-setup.

In Storage Group configuration, set up directories for each of the following: Videos, Trailers, Fanart, Banners, Screenshots, and Coverart.

Optional Step: If you would like to use a combination of Storage Group and locally hosted video, you can do the following. On the frontends, go to Utilities/Setup->Setup->Media Settings->Video Settings->General. Change "Directories that hold videos" to point at a directory that is not the same as the one the Storage Group points at. If the local video setting and the Storage Group setting point at the same path, MythVideo will prefer the Storage Group path and ignore the local one.

Metadata Grabbing with Storage Group Files

Doing a metadata lookup when using Storage Groups will download the images to the Fanart/Banner/Screenshot/Cover Storage Groups on the backend that hosts the video in question, so they must be defined. If they are not defined, the images will be downloaded to the Video Storage Group instead.

Mass Metadata Grabbing

Storage Group Disk Scheduler

The Storage Group Disk Scheduler provides 3 different implementations, which can be selected with the mythtv-setup General setting "Storage Group Disk Scheduler".

Balanced Free Space

The only criteria considered here is free space, so if you have 5 tuners
and 2 filesystems, one on brand new shiney empty disk drive, and 5 things
are recording at the same time, they'll all go to the same disk since it
has the most free space.

Balanced Disk I/O

The only criteria considered here is the 'weight' on the filesystem at
the time. Items included in the weight are in-progress recordings,
playback, commercial flagging, transcoding, etc.. The disk with the
least I/O activity (weight) will get the recording. In this case, with
the same 5 tuners and 2 filesystems mentioned above, you'd have 3 tuners
recording to one filesystem and 2 tuners going to the other, but there
is no preference as to which filesystem gets the 3 and which gets the 2.

Combination

This is the original Storage Group balancing method. This method is no longer the default.
This method uses a combination of free space and disk I/O to determine
where to put the next recording.

Sorting is done using the following criteria:

If fs1 is local and fs2 is not
then if fs1 has less weight use fs1 else use fs2
else if fs1 and fs2 are either both local or both remote
then if fs1 or fs2 has a lower weight, use the one with the lowest weight
else if both have equal weights then use the one with the most free space
else if fs is not local and fs2 is
then if fs1 has less weight use fs1 else use fs2

Weighting

Note: There is some limited ability to change the weighting, but there is no GUI to set them up.

MythTV uses weights to determine what filesystem/directory to record to. The following default values are
used:

The drive with the lowest weight is used first. In a tie, the drive with
the highest amount of free disk space is used first.

Each Storage Group Directory has its own weight, but MythTV is smart enough
to know if several directories are on the same shared filesystem. When
a weight is applied to a directory that is in use, it is applied to all
directories on that filesystem because we are trying to spread the load
out across filesystems, not directories on the same filesystem.

The starting weight of all drives is 0. Local filesystems/directories
are then offset by SGweightLocalStarting, so by default they are then
at -19 because SGweightPerRecording defaults to 10. This makes it so
that local drives are preferred over remote drives. A local drive has
to have an effective weight of 20 before a remote drive will be used to
store a new recording. If you do not have any playback going on, this
would mean that you'd have to have 2 recordings going to a local drive
before a remote drive would get used. If you have 2 local drives and
1 remote drive, you'd have to have 4 recordings going on locally before
a remote drive would get used.

So, if you have less than 4 tuners and 2 local drives, then the default
should already give you what you are looking for. If you have more than
4 tuners or want to guarantee that all recordings go to the local drives
unless they fill up, you can do so using the undocumented SGweightPerDir
setting.

In order to make MythTV not use a filesystem/directory, we need to
artificially inflate the starting weight for that directory. We can do
this by insertting a setting in the database.

The key for the setting is SGweightPerDir:HOSTNAME:DIRECTORY. The
hostname is the hostname that sees the directory. So if the directory
is actually on the fileserver which is server1 but is mounted via NFS on
server2 which is running mythbackend, we'd use server2 here. The
directory is the local path on HOSTNAME, so you'd put /mnt/video or
whatever you use here for the remotely mounted directory.

SGweightPerDir:server2:/mnt/video

The value that we put in this setting will be applied as an offset to
the initial weight for this directory. You can play it safe by setting
this to something large like 99 or 100 or even higher.

To set the setting, use a browser to submit the request (substituting your MythTV backend's hostname and directory, and the desired weight value):

So, now when the Storage Groups scheduling code runs, /mnt/video will
start out with a weight of 0. Since it is a remote drive, the only
offsets that will be initially applied will be the one we specified in
the settings table with the SGweightPerDir setting. Your two local
directories would each end up starting out at -19 and the remote
directory would start out at +99. So unless you have a huge number of
recordings and playbacks going on on each of your local drives, the
remote directory will never be used unless the locals fill up.

If you run mythbackend with the "-v schedule,file" option, you can
see the weights as they are applied and the logs will show you why
MythTV chose one directory over another when determining where to put
the next recording.