Data Distribution Sub-System

Overview

The Data Distribution sub-system receives real-time interferometer data
from the CDS Network Data Server (NDS) and distributes it by way of shared
memory buffers to any Data Monitor Applications that may be running.

Utility programs for
monitoring and maintaining the shared memory control structures.

Each component is described in detail in the following sections.

Shared
Memory Buffer Manager

Overview

The Shared Memory Buffer Manager is distributed
among the data producer and consumer processes. The buffer queues, semaphores,
status information, etc. needed to manage the buffers are stored in the
same shared memory segment as the data (in future versions, the control
data may be moved to a separate partition to allow write access control
of the buffers).

Buffer Allocation

The control area contains
two buffer lists. A free-list contains all buffers that have no
valid data and are therefore available for distribution to a producer for
filling. The full-list contains all buffers with valid data that
are available for (shared) distribution to the consumers. By default a
buffer is allocated to a producer from the free list and added to the full
list when it is distributed by the producer only after it has been read
by at least one consumer. Note that if there are no consumers the buffers
will fill up immediately and the producers will hang attempting to allocate
a buffer. For this reason the first buffer(s) read by a consumer may be
quite old. The default buffer allocation technique may be adapted to the
specific needs by setting the partition allocation mode with any combination
of the following modifiers.

RELBUF Modifier:
A full buffer is released immediately to the free list if there are no
pending requests from a consumer.

SCAVAGE Modifier:
A buffer in the full list that is not in use and is not reserved for any
consumer may be allocated to a producer if no buffers are available on
the free list. In this case, the oldest available full buffer is used.

RQSYNC Modifier:
A buffer is allocated to a producer only if there is a pending request
for a buffer by a consumer.

Consumer Control

Each full buffer has
an associated trigger mask set by the producer when the buffer is filled.
The consumer process may specify a mask of interesting triggers at the
time the consumer registers. Only those buffers which have one or more
trigger bits in common with the consumer's trigger mask will be considered
for distribution to the consumer.

The consumer interface
allows the consumer to reserve any number of buffers. By default no buffers
are reserved for the consumer. When this is the case, each time a consumer
requests a buffer the consumer interface takes the oldest buffer in the
partition's full-list. If there are no buffers in the full-list, the consumer
request fails or the consumer waits for the next available buffer. When
the buffer is released it is marked as having been seen by the consumer
to prevent it from being reread with the next request.

If the consumer requests
that one or more buffers be reserved for it, the (shared) reservation of
the buffer is performed at the time the buffer is distributed by the producer.
The buffer will be reserved for all consumers that have selected the buffer
by the trigger mechanism and have not filled their specified quota of reserved
buffers. The reservation is canceled when the consumer releases the buffer.

Programming Interface

The programming interface consists of three
components:

Shared Memory Interface (LSMP): a base
class that manipulates the shared memory partition, and allocates and releases
resources (memory, semaphores and access rights) used to synchronize the
memory access.

Producer Interface (LSMP_ACCESS):Provides
control access to the shared memory.

The public interface for these classes and
others used in the data distribution subsystem are described in more detail
in the Data Distribution API Document.

Timing

I ran some timing tests of a prototype Buffer
Manager on my Sun WorkStation (a 167 MHz Ultra-1 with 83 MHz memory). The
programs used were compiled with "CC
-O2" . The results of these
tests are summarized in the table below.

Parameter

Value

Producer Overhead

44 us

Consumer Overhead

58 us

Process Swap

56 us

Write data rate

150 MB/s

Read data rate

100 MB/s

The fine print:

The times tabulated here are real-time intervals not CPU-time. I assumed
that this would give a more reliable result since overhead incurred in
context switching may not be reflected in the CPU time. The side effect
of this decision is that the time measurements are susceptible to other
background processes

No process swapping is required when the producer is run alone. When one
or more consumers are running, the rate of process swaps/record is assumed
to be (Ncons+1)/Nbuffer where Ncons is
the number of consumers and Nbuffer is the number of buffers
(in this case 8).

Read and write data rates are probably dominated by the programmed data
transfers. The point here is to demonstrate that there are no major bottlenecks
in writing-to/reading-from the shared memory.

Interface to Frame Packages

The shared memory API has been interfaces
to the two popular frame libraries: FrameCPP, the C++ library written and
supported by LDAS, and FrameL, the C library from Virgo. The FrameCPP interface
is by way of two classes iSMbuf and oSMbuf based on the C++ stream classes
istream and ostream, respectively. These classes are also documented in
the Data Distribution API Document.

The interface to FrameL is by way of a
redefined FrIO structure and
the associated interface functions FrIOClose(),
FrIOOpen(), FrIORead() and
FrIOWrite(). These are defined
in ONLFrIO.h and ONLFriIO.cc.
Note that ONLFrIO is implemented
as a C++ package with a C interface in order to communicate with both FrameL
and LSMP.

The Data Pushers (DpushF,
DpushM, DpushRT)

The data pusher programs receive data from
various data sources and copy them to a shared memory buffer. One or the
other program must be running at all times in order to provide data to
the monitor programs. Several data pushers may be run in parallel on different
partitions to provide multiple data sets to different monitor groups. Care
should be taken not to start more than one Dpush process on a single partition.
as this might cause a monitor to receive duplicate copied of a single data
frame. The three Dpush programs read data from different sources, but otherwise
function in much the same manner. DpushF reads frames from the CDS Network
Data Server (NDS), DpushM receive frames broadcast by the Frame Broadcaster,
and DpushRT replays frames stored on spinnig media in real time. On startup,
Dpush will create a shared memory partition with the specified name, length
and number of buffers if the named partition does not already exist. It
then copies data from the specified source to the shared memory partition.
It also keeps statistics on the number of missing Frames and the average
rate of arrival of the frames and prints them to a log file every 20 minutes.
The syntax for the Dpush command is the following:

The arguments to Dpush
are:<node>
Node name or IP address of the network
data server (DpushF, DpushM).<file>
Input file name or wild-card expresion.
(DpushRT only)<pname>
Name of shared memory partition to
receive the data.<nby>
Minimum buffer length for the partition.<n>
Minimum number of buffers in the partition.<logfile>
Name of file to receive error and statistics
messages.<ticks>
Number of seconds between statistics
summaries.

Entering an invalid argument causes a help
message to be printed out. The Dpush
executables reside in the $DMTHOME/bin
directory.

Using DMTrestart to run DpushF

DMTrestart is a script used to run DpushF
and to insure that it will be restarted if it fails. It is most useful
in testbed situations since the combination of the broadcast version (DpushM)
and procmgt are now the preferred running mode. It performs the following
functions:

Make sure that a copy of DpushF is not already
running.

Check that the NDS server is alive and responds
to a version number request. If not, wait for it to start, testing in 20s
intervals.

Generate a log file name of the form $HOME/logs/DpushF-yyyy.mm.dd-hh.mm.

Start DpushF with all the appropriate arguments.

Wait for the DpushF to fail/terminate.

When running under DMTrestart,
DpushF
can usually be upgraded without affecting any running monitors by replacing
the DpushF
executable and killing the process. It will be restarted with the loss
of at most 1-2 frames, without otherwise affecting the client processes.

Data
Distribution Utility Programs

A set of utility programs is has been written
for maintaining the Shared memory partitions. These are listed below:

smcreate - Create a shared memory partition.

smcreate
is used to create a shared memory partition. The syntax for the smcreate
command is

smcreate
<partition> [-nbuf <n>] [-lbuf <length>]

The parameters to smcreate
are:

<partition>Name of the partition to be created.<n>Number of buffers in partition.<length>Length of buffers.

smdump - Print status of a shared memory partition.

smdump
prints the current status of a shared memory partition. The syntax for
the smdump command is

smdump
[<partition>]

The parameter to smdump
is:

<partition>Name of the partition to be dumped. If
<partition>
is not specified, thepartition name will be taken from the
LIGOSMPART
environment variable.

smraw - Formatted dump of the data from a shared
memory partition.

smraw
is used to produce a formatted dump of the data stream from a shared memory
partition. The syntax for the smraw
command is

smraw <partition>
[-nwmax <nw>]

The parameters to smraw
are:

<partition>Name of the partition to be dumped. If
<partition>is
not specified, thepartition name will be taken from the
LIGOSMPART
environment variable.<nw>
Number of (32-bit) words to be dumped
per record, by default <nw>
is 64.

smrepair - Remove inactive clients.

smrepair
removes inactive clients (i.e. clients for which the registered
process no longer exists) from the client list of a shared memory partition.
All buffer reservations for the deceased consumers are canceled and if
no active consumers remain, all the buffer use counts are set to zero.
In most cases this will free a hung-up partition. The syntax for the smrepair
command is

smrepair
<partition>

The parameter to smrepair
is:

<partition>Name of the partition to be repaired.
If <partition> is
not specified, thepartition name will be taken from the
LIGOSMPART
environment variable.

smstat - Print shared memory partition status
parameter

smstat prints requested parameters
of the shared memory partiton. The parameters which may be interrogated
are:

buffer_tot
Total number of buffers read into partitionfreebuf
Number of buffers in the free listfullbuf
Number of buffers in the full list.last_id
Data ID of most recent buffer.lbuf
Maximum buffer length.maxcons
Maximum number of consumers.name
Partition namenbuf
Number of buffers allocates in the partition.ncons
Number of active consumers.pflags
Partition flag bits.semglobal
Global semaphore ID.shmid
Shared memory partition ID.use_count
Number of processes using the shared memory partition.usedbuf
Number of in-use buffers.version
LSMP version that created the partition.

The command syntax is:

smstat <param1>
[<param2> ...] <partition>

Where <paramx>
is a parameter name from the list, and <partition>
is the name of the shared memory partition to be interrogated.

Last Update: July 25, 2000Please send comment and suggestions to: John
Zweizig