SFDU General Information

The SFDU acronym stands for Standard Formatted Data Unit. An
alternative meaning is Standards For Data Understanding.
SFDU is a set of international standards for packaging, classifying,
and aggregating data into self-identifying data products. A standard
20-byte label begins the entire data product. More similar labels
are attached to data products nested within each original data product. The
SFDU standard has been extended to provide data description languages,
a data dictionary structure, and Control Authority services.

SFDUs were originally proposed by Panel 2 of the CCSDS to solve the
problem of data transfer between heterogeneous environments. The CCSDS
is an international organization made up of representatives of most of
the world's space agencies. Its Panel 2 is developing standards for the
interchange and archiving of data collected in space experiments. This
organization was founded because space scientists around the world realized
that without a standardized way to label and package space data, it would
become unmanageable to archive and retrieve huge amounts of data in many
varieties of data formats. Data sent from one site to another site might
be unreadable or unusable at the second site. The plan is to have
software tools and services available to space scientists and agencies
around the world. This would enable any user anywhere to
unpack, read, and process any space data.
There is the additional intention that these labels be planned in such a way
that software may be written to examine the whole of an SFDU object, find
certain parts of it, and automatically provide services upon or regarding
the information in those certain parts. This embodies what is often referred
to as the requirement that SFDUs shall be "human-readable and machine-
parsible"; in other words, the labels will not be of such size or complexity
as to significantly complicate a scanning by eye of the SFDU's contents, but
the labels will also contain enough information so that software can usefully
dissect the SFDU object.

The SFDU standard includes the description of the activities of specialized
personnel whose work can be broadly characterized as combining that of
librarians, advisors, and administrators. These are the personnel who staff
what are called Control Authority Offices (CAOs). Another term often used for
such an office is Member Agency Control Authority Office (MACAO). The
functions of these offices are:

to assist scientists with the archival and retrieval of data that has been
stored using SFDUs,

to advise and assist exploration projects as they determine how to put
together their SFDUs,

to assist in the creation of subsidiary CAOs ("descendant MACAOs") at such
times as projects or agencies may wish to set them up. For example a
descendant MACAO may be set up to deal with the data of a particular mission,

to assist in the orderly de-activation of descendant MACAOs as this may
occur. This would involve the associated safe transfer of any collected data
to the higher CAO's archives,

to furnish an annual report on SFDU usage and descendant MACAO activity to
the CCSDS, and

to continue to improve and develop the SFDU standard, by means of the
practical experience that is acquired.

The standards that are being developed are worked on in the form of a series
of "books." The development of a particular book is
delegated at first to a small group of professionals involved with SFDUs.
Eventually, as the book is developed, these professionals
put the book out for commentary and approval by the wider community.
This process is associated with the book moving through a series of stages
that have been named with the colors White, Red, and Blue.
White books are under review by the initial technical panel. Red books have
graduated to formal review by the member agencies. Red books, while
recognizably more mature than white books, are still subject to iterations and
revisions. Blue books are the final result of the development. Blue books are
documents that have been approved by all member agencies.

At this writing (March, 1992), the standards have all reached the "Red
Book" stage, except for an explanation of fundamental SFDU structure and
construction rules, which is a Blue Book. Red books are being developed on
additional structure and construction rules, Control Authority Office Policies
and Procedures, and technical issues concerning the computer-parsed aspects of
SFDU information.

MORE DETAIL ON SFDUs AND INFRASTRUCTURE

The primary focus of SFDUs is transfer of information or knowledge
as opposed to the rapid access to the information. To do that, there
is heavy focus on the metadata or documentation that needs to go along
with the data set to make it generally usable and understandable by
the science data recipients.

The SFDU is really comprised of a set of international
standards and their supporting infrastructure.

In the standards area, there are a number of categories of
standards. There are packaging and unpackaging standards, which include
the classifications of data. There are the description languages,
which are used to write documentation that describes the data. There is a
dictionary structure, which is used to document or establish a standard
interchange form for dictionaries. Additionally there are other services
that are provided by this international organization infrastructure.
These are called Control Authorities.

The Control Authorities provide an international registration
process for metadata i.e. documentation about data. There are a number of
coordinated services that result from the fact that they follow standards
in this area.

The main function of the Control Authorities is to register descriptions
and to disseminate those descriptions upon request. The way the desired data
description is identified in these requests
is through a particular identifier for the data. Depending on your local
Control Authority, other services could include the distribution of
software and the checking the metadata for completeness.