Data Acquisition

From its inception, CDIP has considered automated data acquisition
to be a core responsibility. Data is never merely recorded locally,
at the sensor; it always transfered back to a central archive. This
is done at regular intervals, without any operator intervention.
For a program monitoring such a vast spread of locations - in 2003,
for instance, CDIP stations ranged from Guam to Georgia - there are
two distinct challenges in this process. First, data from the
sensors must be organized and stored locally. Second, these data must
be tansferred back to a central location for processing and analysis.
Over the course of its history, CDIP has employed a variety of
hardware and software techonologies to meet these requirements in
a reliable and cost-effective manner.

As noted above, CDIP's data acquisition is a two step process.
The first step is recording the stream of measurements from
the station's sensors on a local computer called a 'shore station'.
This is a constant, non-stop process; the shore station is
continuously linked to the sensors and records their readings -
generally one or more sample per second - 24 hours a day, 365
days a year.

The second step in the process is to transfer the data from the
shore station back to CDIP's central data processing and archiving
facility. This facility, located in La Jolla at SIO, is known as
the Lab. To accomplish this, the Lab's computers contact the shore
stations once or twice per hour, using a modem or network connection
to collect all the accumulated data.

One key aim of the acquisition system is to keep CDIP's data as
up-to-date as possible. Most stations are now contacted every 30
minutes, and the data they return are generally available on the
website within minutes. (All the data on the web appear to be
at least 30 minutes old because the files and products are
labeled with start times, which means that a file running right up to
the current time has a start time of 30 minutes ago.) The shore
stations themselves also feature several display pages with real-time
data straight from the sensors. In many places CDIP's shore stations
are located in the offices of local users like harbor masters and
lifeguards, who can refer to the most current measurements at any
time.

Another key aim of the system is reliability, and to this end the
shore stations maintain a local archive of all data acquired by
their sensors. When the shore-to-Lab link experiences a failure -
i.e. if the network or modem connection goes down - the local
archive can then be used to recover any data which were not
successfully transmitted back to the Lab. Note, however, that
none of CDIP's sensors are equipped to archive data themselves.
This means that if there is a failure on the sensor-to-shore
link - as occasionally occurs because of radio interference -
data are lost and cannot be recovered.

Although the overall setup and operation of CDIP's data acquisition
system has remained remarkably consistent since the mid-1970s,
technological advances have resulted in numerous changes
to the hardware used in the system, both at the shore stations and
in the Lab.

Shore stations

In the 1970s and 1980s, shore stations were composed of custom-designed
circuit boards that operated exclusively under hardware control. Holding
a very limited amount of data, these boards would upload their
contents to the Lab when contacted via modem. In the early 1990s,
more control and flexibility was acheived with the introduction of
the PC-based 'smartstation'. Unlike its hardwired predecessors, the
smartstation was based on a generalized architecture, allowing it to
bring in data from any type of sensor. Equally important, the smartstations
opened up two-way communications between the Lab and the shore stations.
Instead of simply dumping data back to the lab, the smart stations sent
back diagnostic information and could receive and interpret a range of
instructions. This allowed numerous sensor parameters - sample rate,
record length, etc. - to be controlled remotely from the Lab. Another
major advance introduced by the smartstation was the real-time display:
at shore stations where interested users wanted to view the data,
video output displayed sensor reading and derived parameters in real time.

At the end of the 1990s a gradual shift was begun to a new
generation of shore stations. Based on Sun Microsystems' Ultra 5
and Blade 100 computers, these new stations greatly expand upon
the flexibility and control of smartstations. These shore stations'
numerous features include:

full remote shore station access via modem or network links;

a multi-screen, graphical display of all measured parameters;

continuous data archiving with a multi-year capacity;

a seconday, independent zip-based backup data archive; and

a smart UPS that ensures graceful shutdown and startup.

As of 2004, the vast majority of CDIP shore stations are Sun
systems,
although a few smartstations remain in use.

The Lab

The hardware in the Lab is responsible for contacting all of the
shore stations, collecting available data, and archiving it
permanently. When CDIP first started operations, this was all
handled by a Nova minicomputer. The Nova contacted shore stations
with a specially designed modem and communications boards, and
then archived the data on nine-track reel-to-reel tapes. In the
1980s the switch was made to PCs. While the modem and communications
remained largely the same, the x86 PCs in use archived data to
floppy disks in addition to tapes, making the data much easier
to handle. The use of a PC as the Lab's main data acqusition machine
continued until the end of the 1990s, when the switch was made to
a Sun Ultra 10 workstation. The Sun, still in use today, uses
standard modems and network connections to contact the shore stations,
and it archives all the data collected on a hard drive and DLT tapes.

Complementary sets of software are used in the field and in the Lab.
For communications and data transfer, kermit is used on both the
shore stations and lab computers. Customized kermit scripts transfer
data and system information using a number of network and modem
protocols. These scripts provide extensive diagnostic information,
allowing any communcations problems to be quickly and automatically
addressed.

To read data from gauges and manage it locally, the shore stations
use a combination of CDIP-developed FORTRAN and C code. The very
same FORTRAN libraries are used in the Lab to manage and analyze
the data once they come back from the field. For displaying the data
and calculated parameters, the shore stations use MATLAB to generate
a range of tables and plots. Similarly, in the Lab MATLAB is used
for creating a great many of the graphical products found on CDIP's
website.

For more details on the data acquisition software and setup, please
refer to the documents listed below. Any questions regarding the
content of these documents should be directed to the
webmaster.