In a real-world clustered HTTP Streaming scenario, you
want your site to offer robust playback, even in the face of server-side
streaming problems. HTTP Streaming failover, new in FMS 4.5.2, lets
you control how FMS handles certain streaming problems.

Overview of HTTP Streaming failover

HTTP Streaming failover features primarily address the
issues of liveness and dropout.

The following figure illustrates a simple two-server cluster
of packagers with a Reverse Proxy controlling access. The explanations
that follow use this figure as a reference.

Packagers in a two-server cluster

Liveness is a server-side situation in which a
packager advertises a stale bootstrap (that is, a stale view of
a live stream). Although there are other potential scenarios, the
following describes one liveness situation:

The stream
has been running normally for some time on the server side.

Packager 2's connection to the encoder fails. The connection
between Packager 2 and the proxy continues running. Suppose that
at the time of this failure, the last successfully written fragment
on Packager 2 was still #700 with current media time as X.

A short amount of time elapses, in this case, 10 fragment
intervals.

The server-side now suffers from liveness. Packager 2 is
serving stale bootstrap information. Packager 2 indicates that there
has been no update in the bootstrap after time X, even though more
recent fragments are available on Packager 1 (that is, the current
media time is greater than X.)

Playback stalls at the player end.

Dropout - is a server-side situation in which a packager
has gaps in its bootstrap (that is, gaps in its fragment list).
The following scenario describes a dropout situation:

The stream has been running normally for some time on the
server side.

Packager 2's connection to the encoder fails. The connection
between Packager 2 and the proxy continues running. Suppose that
at the time of this failure, the last successfully written fragment
on Packager 2 was still #700.

A short amount of time elapses, in this case, 10 fragment
intervals.

Packager 2's encoder connection is restored and remains restored
for a long time.

The server-side now suffers from dropout. Packager 2's bootstrap
is missing entries for #701-710, even though those fragments are
still available on Packager 1.

Playback stalls at the player end.

HTTP Streaming failover provides a two-part solution:

Best-effort fetch

Control plane management

Enable best-effort fetch

Best-effort fetch enables the OSMF and iOS video players
to continue playback as normally as possible in the presence of
short-term liveness and dropout problems on the server-side.

You enable best-effort fetch by adding and configuring a <bestEffortFetchInfo> element
in the manifest.xml file on the server side, as as illustrated in
the following:

No client-side configuration is required. For more information,
see Manifest.xml.

OSMF

The OSMF 2.0 player adds client-side
robustness by supporting best-effort fetch. Specifically, when best-effort
fetch is enabled on the server, OSMF 2.0 attempts fetches for fragments
that have not been advertised in the bootstrap, but are expected
to be present.

For more information on OSMF 2.0 client side
functionality, see the OSMF documentation on SourceForge.

HLS

For iOS Video players, FMS 4.5.2 enables
best-effort fetch for HLS through settings in the <bestEffortFetchInfo> element
in the manifest.xml file.

Configure HTTP Streaming failover server settings

You configure server settings for HTTP Streaming failover
in the httpd.conf and manifest.xml files.

Note: Best-effort fetch relies on configuring the back-end reverse-proxy
with HTTP 503 failover, meaning that when the reverse proxy receives
a 503 response from a packager, it retries the request with the
next packager in the pool. If all packagers generate a 503, the
proxy generates an error that it caches for less than a fragment interval.
You can configure the HTTP error code, as described in Configure
live and on-demand HTTP Streaming at the server level (httpd.conf).

Code the control plane application

To implement HTTP Streaming failover, you write a client
application that manages the state of events and streams by using
a set of REST-based control plane APIs. Control plane is a router
term and in effect, that is what your client application does through
these APIs.

The basic workflow is as follows:

Begin repeatedly checking stream status at a set interval.
The exact interval depends on your configuration.

Disable any stream with a bad status code.

You could also compare the age of the bootstrap file and
disable the stream for which the bootstrap is older than some predetermined
interval.

FMS control plane REST interface request path

The
FMS control plane REST interface request path differs, depending
on whether you are using HDS, HLS with adaptive bit-rate (ABR),
or HLS with a single bit-rate stream:

HLS with single bit-rate: http://www.example-server.com/ctrlplane/livepkgr/_definst_/liveevent/livestream.m3u8/

You
use HTTP GET or POST, depending on the endpoint and parameters passed.

Query stream status

Description

Returns stream status.

Endpoint

status

Result

If the .disabled file exists in
the streams directory for the respective associated stream, the
control plane module returns "disabled" in the status element for
the particular stream. This endpoint returns stream age for each
stream (multiple streams in case of MBR) calculated as current time in minutes - last modified time for
the bootstrap file.The up element indicates whether
the stream is up. It is set to false if the bootstrap
has not been updated in specified number of seconds (configurable
as MaxBootstrapAge). The global up element
is set to true if all bootstrap up elements are
true, otherwise it is set to false. The endpoint sets the upper
level status to disabled if all
streams are disabled, otherwise it is set to enabled.

Disable an event

Description

Disables all streams
for an event (works for both HDS and HLS streams). You send this
as an HTTP POST request with no request body.

Endpoint

disable

Result

Upon receiving this request, the
control plane module creates a .disabled file in
all streams directories (multiple streams directories for an MBR
stream) and returns an XML formatted response indicating success
or failure for each stream.

If a client subsequently requests
streams for the event, the Apache module (mod_f4fhttp or mod_hlshttp)
checks for the existence of a .disabled file before
processing the request. If a .disabled file exists,
the module returns a 503 error (configurable in httpd.conf).

Disable a stream

Description

Disables a stream (works
for both HDS and HLS streams). You send this as an HTTP POST request
with the stream name specified in the body.

Disable a stream
if you detect a liveness problems or to take a faulty packager out of
production for some other reason. As mentioned below, the Apache
module returns a 503 error, making the faulty packager a non-entity.

Endpoint

disable

Result

Upon receiving this request, the
control plane module creates a .disabled file in
the streams directory and returns an XML formatted response indicating success
or failure.

If a client subsequently requests the stream,
the Apache module (mod_f4fhttp or mod_hlshttp) checks for the existence
of a .disabled file before processing the request.
If a .disabled file exists, the module returns
a 503 error (configurable in httpd.conf).

Enable an event

Description

Enables all streams
for an event (works for both HDS and HLS streams). You send this
as an HTTP POST request with no body.

Endpoint

enable

Result

Upon receiving this request, the
control plane module deletes any existing .disabled files
in all streams directories (multiple streams directories for an MBR
stream) and returns an XML formatted response indicating success
or failure for each stream.

If a client subsequently requests
streams for the event, the Apache module (mod_f4fhttp or mod_hlshttp)
processes the request.

Mark an event as done

Description

Signals that an event
is complete. You send this as an HTTP POST request with no body.

You
signal an event as done so that the client is notified when content
ends. This achieves two things: the control plane does not activate
failover, and client applications can take appropriate action to
signal the end of content to the user.

Endpoint

done

Result

Upon receiving this request, the
control plane module creates a .done file in all streams
directories (multiple streams directories for an MBR stream) and
returns an XML formatted response indicating success or failure
for each stream.

Mark a stream as done

Description

Signals that a stream
complete. You send this as an HTTP POST request with the stream
name specified in the body.

You signal a stream as complete
so that the client is notified when content ends. This achieves
two things: the control plane does not activate failover, and client applications
can take appropriate action to signal the end of content to the
user.

Endpoint

done

Result

Upon receiving this request, the
control plane module creates a .done file in the associated
streams directory and returns an XML formatted response indicating success
or failure.

Mark an event as in-progress

Description

Signals that an event
is still in progress. You send this as an HTTP POST request with
no body. This request undoes a /done request.

Endpoint

inProgress

Result

Upon receiving this request, the
control plane module deletes any existing .done files
in all streams directories (multiple streams directories for an
MBR stream) and returns an XML formatted response indicating success
or failure for the stream.

Mark a stream as in-progress

Description

Signals that a stream
is still in progress. You send this as an HTTP POST request with
the stream name specified in the body. This request undoes a /done request.

Endpoint

inProgress

Result

Upon receiving this request, the
control plane module deletes any existing .done file
in the associated streams directory and returns an XML formatted response
indicating success or failure for each stream.