HOW TO: Build redundant playout system with OnTheAir Node

It is possible to configure a Node with Master and Slave mode. In this mode one Node (the slave one) will follow automatically everything that the other Node (the master one) is doing. So:

with OnTheAir Live, if you send a playlist and start to play it on the "Master" Node, the "Slave" Node will follow.

with OnTheAir Manager, if you send a schedule to the Master Node, the "Slave" Node will download the same schedule and use it.

So it is very easy to create a fully redundant system, and you can read more about the master and slave mode in this article. But there are a few things to know when talking about redundancy, see below for hardware and workflow considerations.

General Considerations

Questions that should be answered are:

Automatic or manual ?

Using a "shared" or "redundant" storage?

Are animated graphics are required or not?.

Should we play the same thing at the same time on the redundant system, or play something different?

No list will be completely comprehensive, but we'll try to be as complete as possible.

Automatic or manual ?

The first question to answer is wether you want

almost no downtime: if an issue happens on one computer, the backup system should take over automatically. This is of course what everybody would want, but it implies some investments such as:

having 2 complete sets of licenses (if you have one OnTheAir Node and OnTheAir CG, you will need 2 licenses of each). Note that you will need only one license of OnTheAir Manager.

having everything powered, cabled.

having an automatic change over system

manual: if an issue happens on one computer, an operator will do some operations to be back online. It can be very simple and quick operations such as moving the dongle from one computer to another (so you only purchase one license), or a bit more operations, such as taking out the faulty computer and replacing it with a new one that was already configured.

When building a redundant system with Softron products, usually (but see above for some comments on this) you have 2 different computers playing the same clips at the same time (check how to configure OnTheAir Node for that). But if there is an issue happening, OnTheAir Node will not monitor the signal to check if it is still playing out and change from one to the other. For that you can of course do it manually: connect the two outputs to your in house video router. When one computer fail, you manually switch to the other output.

But the most automatic solution is to use an an "automatic change-over" system. What you will do is connect the 2 outputs (directly or through a video router) to the "automatic change-over". These systems analyse the signal getting in the primary input. If they detect an issue on that input, they will automatically switch to the secondary input.

When choosing for an automatic change-ver, you must make sure of a few things:

Does it automatically switch if the signal is black? Some change-over only switch if the detect not signal at all. But in some occasions, you may still have a valid signal coming in, but black, which is not what you would like of course.

Check also if it has all the functionality that you need. If you are working with RS232 communications, you may want to opt for a changeover that has RS232 connections too.

Can it be manually switched from one position to the other?

Can it be switched with GPI from one position to the other?

Here are a few models that can handle more or less of this at different price point:

Note again that we can not endorse those products, and we would recommend working with an integrator that will be able to advise you on the best model for your use.

Shared or redundant storage:

The primary concern is that the storage must always deliver a constant bit rate to ensure consistent playout. While Softron does not recommend any specific storage solution (we recommend working closely with an integration specialist to select a storage solution that meets all of the users needs) we like to point out that media can be read (played out from) a variety of sources: a local hard drive (such as an internal hard drive,) an external Thunderbolt, USB 3.0, a NAS or a SAN. It is all up to the individual user.

However, there are two main ways of handling storage when using a redundant system:

Redundant storage:

Each playout station has its own storage from which it reads the files.

Pros: This configuration is the safest because if one drive or drive system fails, the other can take over.

Cons: Each new file must be copied to the two locations in the same hierarchy. With a MAM system this can be easily done, but if done manually one may be missed.

Shared storage:

The two playout stations read the files from the same shared storage.

Pros: You do not have to handle two different locations.

Cons: There is no redundancy. If the storage fails, it fails and there is no way to retrieve the data. Therefore, you must prepare a backup plan for the shared storage which will also increase the price. Additionally this is more limiting because you are you can use only a NAS or a SAN.

Playing the same thing at the same time ?

The other question that comes around a lot is wether the 2 Nodes should play the same thing at the same time, or play something different. It is true that if the two Nodes are playing the same clips at the same time, and if there is a corrupted clip, the issue will happen on the 2 Nodes at the same time. In that case, playing the same thing at the same time may not always be a good idea for redundancy.

If you are concerned about that, you have other options:

Play something different on the backup Node, like a playlist in Loop. This would imply to have a manual control in order to send the new playlist.

If we are talking about a scheduled playout, you could just change the clock of the redundant Mac so that it plays all the events and clips one hour earlier. You can than monitor that signal and if you see something happening on that output, you have one hour to fix it on the main output.

Of course if you decide not to play the same thing at the same time, you will not need to use the "Master/Slave" mode of OnTheAir Node.

With OnTheAir Manager, have a backup copy of the database !

If you are using OnTheAir Manager, make sure that you back up the database regularly, both locally and to another storage. Indeed, the OnTheAir Manager database stores all the schedules, playlists, etc, so if you loose it or it gets corrupted, you will have to do all your schedules again. Check this article for recommendations on where to store and how to backup that database: https://softron.zendesk.com/hc/en-us/articles/216279558-Recommendations-about-the-database-BACKUP-regularly

About the licenses: Dongle or Serial?

In general, we recommend using dongles, it saves you all the activation and deactivation process, and it is thus very easy and quick to move a license from one computer to another.

Also, with a redundant system it allows you to have just one license and move it from one computer to another easily in case an issue happens (this of course won't work with an automatic redundant system)

Case 1: Playout without Graphics

1.1 The simplest configuration

Whether it is for OnTheAir Manager with scheduled playout or for OnTheAir Live with unattended playout, if you are looking for a simple and affordable playout solution for a single channel output with redundancy, here is what you could do:

Configuration:

Only one of the 2 Mac has

a license for OnTheAir Node, it is on a dongle so it can be quickly and easily moved

a video device

The other Mac is just ready to take over in case the first one has an issue. When this happens, the user will have to take the dongle and video device from the first device, and connect it to

1.2. Having a system ready for air, but manually change it

Configuration:

Here each Mac is "ready" and have its own hardware, cabled to a video router.

you have only one license on a dongle connected to the "Master Node".

The "Slave Node" is playing the video files, but in demo mode (it has the blue frames)

if the master node has an issue, you will have to:

move the dongle to the "Slave Node"

change the video router so you now send the output of the slave Node to air

1.3. With an automatic changeover system

Configuration:

In this setup, you have 2 complete systems, each with its own license

Additionally there is an automatic changeover system

In case there is an issue with the "Master Node" and the output goes to black or freezes, then the automatic changeover will detect it and automatically switch to the "Slave Node".

1.4. With an additional computer to control the Nodes

Configuration:

Here we have just added one more computer that will control the OnTheAir Node. This computer will run either OnTheAir Manager or OnTheAir Live. If this computer fails, playback will still continue on OnTheAir Node.

Case 2: Playout with Animated Graphics

If you want to overlay dynamic graphics with OnTheAir Node, you need to add a license for OnTheAir CG. Check here if you want to know how to control OnTheAir CG from OnTheAir Node: https://softron.zendesk.com/hc/en-us/articles/214727528-HOW-TO-Control-OnTheAir-CG-Server-from-OnTheAir-Node

Otherwise the configurations and choices will be roughly the same as the ones above.

On a Mac mini with the direct link

Below is a diagram of a configuration using OnTheAir Node and OnTheAir CG on one Mac mini.

Note that to use OnTheAir Node and OnTheAir CG on one Mac mini there are a few things to know:

because the Mac mini uses a small graphics card, performances will be limited for OnTheAir CG. So first test with a demo version and your projects and see if the graphics card sustain what you want.

it is recommended to disable the "Smart Resize and Graphics" option in the playback preferences of OnTheAir Node, this will ease up the use of the graphics card by OnTheAir Node.

OnTheAir Node will send the video frames to OnTheAir CG Server through the direct link and it is OnTheAir CG Server that outputs to the video device with video and graphics: