OFJ Field Journal from Steven Collins - 11/21/95

It's Tuesday the 21st of November. I'm writing this from the MSA. The MSA
(Mission Support Area) is the "control room" for the Galileo spacecraft.
From here we send commands and monitor the telemetry from critical events
on the spacecraft.

It's a smallish room at the corner of the 5th floor of building 264
here at JPL in Pasadena. Out one set of windows is a view of JPL's huge,
cylindrical Space Simulator where Galileo was tested before beginning
its long journey to Jupiter. Out the other windows is a hazy but beautiful
view of the San Gabriel mountains, a great place to ride mountain bikes
(but you had better like to *climb*). A third window provides a view of
*us* from the Viewing Room. The managers sit in there to keep an eye on
things when a "Big Event" (tm) is taking place.

On two walls of the MSA, a pair of digital clocks with big, red letters
show the current day and time: UTC 325/23:33:27. A third clock counts
down the time till Jupiter Orbit Insertion (JOI): JOI -16/01:46:00. Yikes!
only sixteen days left!!

The room is arranged as a set of computer displays and work areas for
different groups of engineers. A little blue sign hangs over each area:
Telecom , Systems, SFP (system fault protection), CDS (command and data
subsystem), Mission Control. Each station has a computer with a pair of
screens to display telemetry and voice-net terminal we can use to talk
to each other. When we need to say something official like, "Roger Systems,
you are go to send command package "X-ray Zulu Three Four Niner."

The sign over *my* head reads AACS (Attitude and Articulation Control
System). I'm here today to watch the final test of our troublesome tape
recorder before JOI. The tape recorder is really part of the CDS, and
normally an AACS guy like me wouldn't be involved, but when the recorder
anomaly occurred, I volunteered to help the CDS unit and the anomaly team
collect and plot the recorder data.

To be honest, I sort of crammed my help down their throats. I have a
lot of experience using the telemetry system and have spent many hours
searching through old flight data trying to solve problems for AACS, so
I just started plotting up recorder data and giving it to people. My plots
seemed to be helping folks and pretty soon I found myself being invited
to anomaly team meetings and given work to do. When a major problem like
this comes up, everyone on the team contributes where they can.

For a couple of weeks after the anomaly, everyone on the team was working
7 days a week and late into the night to try and figure out what was going
on. It was *very* exciting to be involved. It's sort of like trying to
solve a Sherlock Holmes mystery, where you have to find pieces of evidence
and then fit them together into a theory about what happened and how to
fix things.

Sometimes you get false clues and they make you pick the wrong "suspect".
For example, we have a computerized simulation of the spacecraft that
we call "The Testbed". It's built out of our spare flight computers and
hardware. We use it to test command sequences on the ground before we
send them to the spacecraft and to help us figure out problems.

Now get this: on the *very same day* that the spacecraft recorder anomaly
happened, the spare recorder in our testbed also failed! People were saying
that it was a million to one odds that both recorders would fail on the
same day, so our first "prime suspect" was some kind of problem with the
computer software that controls the tape recorder.

After looking much closer at the telemetry from the flight recorder
and eventually taking the testbed recorder apart and looking inside, it
is now clear that the two failures were really unrelated! The software
is fine and we just beat the odds and had two independent hardware failures
on the same day!!!

Anyway, back to the MSA. By getting involved in the anomaly, I learned
a lot about how the recorder works and build some tools for decoding and
plotting recorder data. That's what I'm doing today. We are doing our
fifth test of the recorder, running it for a couple of seconds to make
sure that it still works and that the tape isn't getting stuck to anything.

My job is to collect some data on the recorder and plot it up so the
real tape recorder experts can make sure everything is working ok. This
is our last chance to check on it before we use the recorder to save the
science data coming up from the Galileo Probe on 7 Dec.

The data I want is coming down as a MRO (Memory Read Out). That's where
the spacecraft just sends down a copy of a little section of its computer
memory instead of its normal stream of telemetry channels. At my workstation
under the blue AACS sign, I set start up a program to capture and save
the MRO data as they come down.

There's the data, right on time. On my screen can watch the data being
printed out as I type this. It looks kind of like this:

It takes a while for all the data I need to come down, maybe 15 minutes.
When I have it all, I copy it to a file and run my little program on it.
It slices it up, moves it around, figures out what data happened when. Finally,
it makes a set of plots that I can hear dropping out of the printer. As
far as I can tell, everything looks just fine. But that's for the real recorder
experts to decide. I call the beeper number I've been given and Greg Levanas
(our version of Sherlock Holmes) responds, He says he'll be up in a few
minutes to have a look.

I use the time to Xerox copies of the plots for him and finish this
report.