User Agent and Assistive Technology Support Notes

Description

The objective of this technique is to use text captioning that comes
from a separate text file, synchronize the captions with the media
in a Silverlight MediaElement, and display the captions
in Silverlight.

There are two variations of the general theme of implementing Silverlight
media player controls to work with external timed text: using built-in
capabilities of the Microsoft Expression Encoder tool, and using parsing
code that consumes caption as a raw file format and converts that format
into the Silverlight API's TimelineMarkers representation.
This technique primarily addresses how to use the Expression Encoder
technique, along with media player templates that are also provided
by Expression Encoder or related Silverlight SDKs such as the Smooth
Streaming SDK.

At a pure architecture level, Silverlight uses the TimelineMarkers API
to display caption text at synchronized times. The Expression Encoder
and Expression Blend tools provide a front end to drive the TimelineMarkers API
for the supported formats. The intention of the architecture is so
that Silverlight as a run-time has a base architecture that could potentially
use any existing or future timed text format, but the tools for Silverlight
(rather than built-in features of the runtime) provide the optimized
workflow for producing captioned media projects.

A procedure for using the Expression Encoder and Expression Blend
tools to produce a Silverlight media player application that can consume
timed text in TTML format is provided as Example 1 in this technique.
(Note: prior to the approval of TTML by W3C, DFXP was a format that
used much of the eventual TTML vocabulary. In tools that predate TTML,
this format is often identified as DFXP.)

A purely code-based parsing technique, with the goal of avoiding Expression
Encoder dependencies, is necessarily more complex. This is because
there are multiple issues to solve:

Obtaining and validating the timed text file

Parsing the format into general information items like timings,
text, format etc. that are either consumable directly in a Silverlight
API, or useful as intermediates

Using the timecode information to create TimelineMarker representations
for each timed text entity

Associating the TimelineMarkers with media loaded
by the player

Finding a place to store the additional formatting that is conveyed,
including the text and any formatting

Handling events from TimelineMarkers in the media
player, in such a way that text and formatting behavior can be retrieved
and presented in the Text part of the player UI

Text Captioning Formats

There are several existing text-based formats that are used for text
captioning of prerecorded media. The following are supported as formats
if using the Expression Encoder tool as shown in Example 1 (where the
Expression Encoder generated Silverlight application uses the existing
Silverlight MediaPlayerTemplate and the TimedTextLibrary component.)
For more information on which timed text formats can be referenced
in an Expression Encoder project, see About
Captioning (Expression documentation on MSDN).

SAMI (Synchronized Accessible Media Interchange). For more information
on this format, see Understanding
SAMI 1.0 on MSDN.

TTML (previously known as DFXP). This is the Timed Text format
used by Adobe Flash for its FLVPlaybackCaptioning component,
and is produced by a variety of tools and full-service captioning
vendors. For more information, see Captions on
the Adobe site.

The following are not supported directly in Expression Encoder templates.
To use these formats, application authors would have to write parsing
logic, as shown in Example 2:

Other formats that have not realized wide adoption, for example
Universal Subtitle Format.

In addition to these formats, other formats for device-specific
recorded media (such as DVD encoded tracks) could be cross-purposed
for use by streaming/online media.

Rather than build the parsing logic for all these formats into the
Silverlight runtime, or choosing just one of these formats to support,
the Silverlight design for text captioning at the core level instead
makes the TimelineMarkers property of a MediaElement writeable,
independently of the value of Source. The format of
each TimelineMarker in the collection is very simple:
a time value expressed in the format, and the text value of the text
for that synchronized caption. The time format for TimelineMarkers is
documented as TimelineMarker
reference on MSDN. Converting timed text formats to TimelineMarkers form
as consumed by the Silverlight core can be done by following any of
the following application design patterns:

Authoring the project using Expression Encoder, and using the
Expression MediaPlayerTemplate as the media player UI. In this case,
Expression produces a Silverlight application that includes assemblies
that are generated from code templates. The default build of the
project provides a working library that handles all tasks related
to timed-text format conversion, from the formats as documented at About
Captioning (Expression documentation on MSDN).

The templates of an Expression Encoder project can also be edited,
either editing the XAML for the UI by altering the template, or by
altering the C# code files that define various aspects of the media
player logic, including the timed text format parsers. Then the project
can be rebuilt using the desired customizations. Using this technique,
it is possible to adapt the code to support timed text formats that
are not directly supported in the Expression Encoder project UI.

Using a 3rd party media player implementation that also includes
a codebase for processing timed text formats, producing TimelineMarkers,
and displaying the captions in the player-specific UI.

Including a library that handles just the format parsing, and
using APIs of this library as part of the Silverlight application
code-behind.

Writing all logic that is necessary for timed text parsing AND
application UI display, and including it all in the main Silverlight
application library.

Examples

Example 1: Using Expression Encoder and Expression Blend to produce
a Silverlight media player project from tool output and templates

By far the simplest technique for incorporating existing timed-text
information is to use Microsoft Expression Encoder and the media player
templates that an Expression Encoder project produces by default. You
can use timed text in any of the following formats: DFXP, SAMI, SRT,
SUB, and LRC. Associating a caption file with a media source is done
as an "import" operation in the Expression Encoder tool.
However, the "import" does not necessarily mean that the
timed text file is integrated into the media stream; Silverlight authors have the option
to preserve the file separation. This is useful if the application is obtaining
timed text from a third party source in real-time, or if Silverlight authors have a
production pipeline that makes it difficult to have the captioning
ready in time to be encoded in the stream along with the audio-visual
source files. For third-party timed text files that are served directly
from the third party's HTTP servers, it can be useful to supply a dummy
URL in the initial project encoding. The output of the Expression Encoder
project parameterizes many of the project settings at the HTML level.
This makes it possible to change the URL at any time prior to deployment
without having to rebuild the project. The following code is the HTML
output of a sample Expression Encoder project. Note the CaptionSources
node in the initparams; that is the information item that informs the
Expression Encoder templates where to find the timed text file.

The templates include a library that handles any parsing requirements
for the chosen timed text format, both at the level of getting the
basic text and timing into the TimelineMarkers used
by the run-time MediaElement, and for preserving
a subset of format information that can reasonably be crossmapped
from the formatting paradigm of the source (typically HTML/CSS) into
the Silverlight text object model of the text element that displays
the captions in the running Silverlight application.

The following is a brief description of the procedure for creating
a project that incorporates a separate timed text file.

From the initial Expression Encoder screen, select New Project
from the File menu.

In the Load a new project dialog, select Silverlight
Project.

From the File menu, select Import. Choose the primary media source
file the project will use.

In the Text tab, find the listing for the media source file. Click
the + icon to the right of the file name. This opens
a file dialog.

Choose a timed text file to add to the project.

Build the project. By default the project produces a complete
output folder. The folder includes the media player template XAP,
the timed text file as a separate file, and additional libraries
and XAPs that support the basic template and/or the timed text capabilities.

Optionally, use the features in the Templates tab of Expression
Encoder to generate a template copy. You can edit the template copy
in other tools such as Expression Blend or Visual Studio, to change
the layout or behavior from the default media player template. Template
editing can address requirements such as applying a particular branding
or "look" to the player user interface.

The following is a screenshot of the Expression Encoder (version 4)
interface. The + icon mentioned in Step 4 is highlighted
in this screenshot with a red diamond. The Templates tab
mentioned in Step 7 is on the right side, top-middle. Note that all
tabs of an Expression user interface are dockable; the orientations
shown here are the default, but could be in different locations on
any given computer or configuration.

This example defines a very simple media player class that includes
a display surface, basic controls, and a text display for captions
as part of its default template. The usage code for this control in
XAML is simple, but only because the majority of the implementation
is present in the definition of the media player class.

Note the attributes CaptionUri and SimpleMediaPlayerWithTT. Each
of these is a custom property of the media control class TTReader.
CaptionUri in particular references a URL, in this case a local URL
from the same server that serves the Silverlight XAP. The timed text
file could come from a different server also, but comes from a local
server in this example to conform to the behavior of the test file.

The following is the generic.xaml default template for the media player
control. The template is mainly relevant for showing the named elements
that are shown in the initialization code.

The following is the logic that is particular to obtaining the separate
caption file. Some of this logic is referenced in the preceding template-specific
event handlers. This example uses the asynchronous WebClient technique
to request the file result of the CaptionUri. Make
sure to use AutoPlay=false or some other means to allow
time for the caption file to download before attempting to play the
media file.

The actual parsing can be done using a combination of the "XML
to Linq" facilities of an optional Silverlight library, and
standard .NET Framework string format APIs from the Silverlight core.
An implementation is NOT provided here, due to length considerations.
TTML supports a number of profiles and capabilities. The basic pattern
to follow in the implementation is to obtain the necessary text and
timing information, and to pass it to a function that might resemble
the following code template. This code template takes the raw information,
generates a new TimelineMarker, and adds it to the
collection assigned to the active MediaElement as
identified by "player" in the application.

public void AddMediaMarker(string time, string type, string data)
{
TimelineMarker marker = new TimelineMarker();
marker.Time = new TimeSpan(0,0,(Convert.ToInt32(time.Trim())/1000));
// this logic could vary depending on how time is formatted in the input string; this one assumes raw milliseconds
marker.Type = type;
marker.Text = data.Trim();
player.Markers.Add(marker);
}

Resources

Accessible
Media Project - a reference implementation MediaPlayer control
from the Silverlight product team that includes several accessibility
features including captioning; note that the codebase might not be
updated to Silverlight version 4

Related Techniques

Tests

Procedure

Using a browser that supports Silverlight, open an HTML page that
references a Silverlight application through an object tag. That
application plays media that is expected to have text captioning.

Check that the text area in the textbox shows captions for the
media, and that the captions synchronize with media in an expected
way.

Expected Results

#2 is true.

If this is a sufficient technique for a success criterion, failing this test procedure does not necessarily mean that the success criterion has not been satisfied in some other way, only that this technique has not been successfully implemented and can not be used to claim conformance.