Hardware Failure and Software

Before his lunar landing flight, Jim Lovell said that he planned to land the LM in fully automatic mode. He never got a chance to try. The Apollo 13 story has become among the best-known aspects of the program, thanks to a number of books and a popular movie. Of course, that flight made no attempt at a lunar landing, so it will not be part of our analysis here. The crises during that mission generated plenty of interesting human-machine interactions, however, and the guidance system proved both redundant and flexible in ways that were critical to saving the lives of the crew.

Apollo 14 came ten months after the near disaster, more than a year after the previous landing. In LMP Ed Mitchell's words, ''Apollo 14 had to be a flawless mission.''20 Mitchell had originally been on the Apollo 13 prime crew and the Apollo 10 backup crew. Hence he had a great deal of training, having been through numerous LM procedures, and having worked with Grumman at Bethpage, New York, building the spacecraft. Apollo 14 Commander Alan Shepard, now forty-seven years old and the only of the original Mercury Seven astronauts to fly to the moon, was a pilot's pilot—old school, hands on, stick and rudder, and an ego to match.

About four hours before the planned PDI burn, Shepard and Mitchell checked out the LM. Everything was normal, until about fifteen minutes before the LM, nicknamed Antares, was scheduled to disappear behind the moon and lose contact with Houston.

For all of the Apollo missions, ground controllers spent the long hours of the flights staring at lists of numbers on computer screens as they downloaded the telemetry in real-time. The controllers became exquisitely sensitive to the binary bits that made up the numbers, which appeared as numerical hash to an untrained observer. Lights on their consoles reflected these bits, signaling ''flags'' that represented various conditions on board the spacecraft. Now, as PDI burn time approached for Apollo 14, they noticed something strange. Amid the hundreds of bytes of data, a bit was set at 1 that should have been a 0. It was the abort bit, indicating that somewhere along the line the computer had gotten an indication to abort the landing. The erroneous bit was harmless at this point, because the LGC in the LM was not running the descent program, so it was not looking for an abort input. But once PDI occurred and P63, the braking phase program, began, an uncommanded abort would be a serious problem.

In the real world of electromechanical devices, things can be ''flaky.'' Houston treated the problem this way, asking Mitchell to press the ABORT button, and then the STOP button. He pressed, and reset both buttons. The faulty indication went away.

The crew twiddled around with antennas and gain controls, trying to get the communications links solid and noise-free. Never an easy job.

About forty-five minutes later, Antares reappeared from behind the moon. Everything looked good. But half an hour later, the abort bit set again. Houston mentioned the discrepancy to the astronauts, noting that they were working on a fix to get rid of the problem after the PDI. A little more than two hours remained before the critical burn.

Houston: ''And, while we've got that display up, Ed, could you tap on the panel around the ABORT pushbutton and see if we can shake something loose?''

Mitchell complied: Tap, tap. Again the error went away.

Houston: ''OK. Antares, we'd like to kind of sit here a minute and watch it.'' Ground controllers uploaded a navigation vector into the computer. The astronauts spotted their landing site on the moon as they passed over, on this the last orbit before landing.

The radio conversation was calm and matter of fact, but everyone knew the story: the flaky abort button was a showstopper. Without some kind of fix, a patch, even a ''kludge,'' the landing could not continue.

In Houston, and in Cambridge, engineers like Don Eyles were working furiously. Jack Garman, who had been in the back room during the Apollo 11 program alarms, remembered the intense pressure of only having two hours to solve the problem: ''The worst nightmare of all.''21

Unaware of the feverish activity on the ground, Mitchell became concerned. ''Do you think we're going to come up with something on this problem with the ABORT button?'' he asked Houston.

''We're working it right now,'' EECOM Fred Haise replied, ''and also MIT's working it. Needless to say, we're busy here, but we think we got a solution.''

''Good enough. Something—is it something like a solder ball?'' Mitchell's early guess was exactly right; according to the post-flight analysis, it was indeed a ball of solder in the switch, jarred loose by the flight, floating around in zero gravity and causing electrical mischief. Several similar switches were later x-rayed and found to have similar problems.22

''Well, we don't know yet,'' Houston replied. ''We got about 19 minutes until loss of signal here, so we'll have something to you before then, and we'll have some time to pick it up on the other side.''

The bit came back again. Tap went Mitchell. Out again.

Ground now had some ominous commentary: ''The implications of that bit being set, I guess you also realize, means that in 63 we're going to find ourselves in P70.''

That was Apollo computer lingo for disaster. Translation: when we're in the braking phase (P63), we could instantaneously go into an uncommanded abort (P70). If this were to happen when the LM was on its way down, it would incorrectly abort the mission—at best a massive disappointment, at worst a dangerous surprise.

Shepard was nervous and felt powerless to fix anything. ''You can't climb into a computer buried deep inside a spaceship with a screwdriver or a wrench,'' he later wrote. The familiar mechanical world of nuts and bolts was long gone.23

Finally Eyles came up with a clever solution. ''As soon as he identified it,'' remembered Garman, ''everybody went, 'Yep, that's it.''' Houston tested the procedure and then called it up to Apollo 14.24

Mitchell wrote the procedure down on his timeline for the PDI: ''Verb 25 Noun 7 Enter;105 Enter;400 enter;0 enter.'' The procedure was complex, and touchy. If the bit set before a certain command, there would be problems.25

That was the last Houston heard from the LM before it went around the moon for the final time. About forty-five minutes later, the LM emerged. As soon as it came in on the radio, Houston sent another procedure. It turned out the one they'd prepared before had some time-critical moments, so they scratched it and came up with a new one.

The ground controllers could have uploaded the procedure automatically, through the digital telemetry link. But instead they called this complex series of commands up to the astronauts by voice, had them write it down, and then key it in by hand. To this day, the computer's designer Eldon Hall believes this meant that the flight crews didn't really trust the computer. While potentially more error-prone, the manual entry gave the people in the loop the confidence that they knew what was being entered.

This fix has often been described as a program change, which is incorrect. Actually, it was a kind of bit-level hack right into the main registers of the computer. The procedure involved about seven steps. An initial DSKY command fooled the computer into thinking it was already executing an abort, so it would not respond to the stuck button. This prevented the computer from checking the abort bit, but it also inhibited the automatic commands for throttle up, the automatic guidance steering, and processing of the landing radar data. Thus the astronauts would manually throttle up for PDI, enter some DSKY commands to start the computer's descent guidance, restore some of the mode flags in the computer, and then command the computer to decrease the throttle when the burn ended.26 Because of these changes, the computer could no longer accomplish an abort with the button, so the crew would have to initiate it themselves by key presses on the computer, separating the stages and going through number of other steps.27 As with other Apollo procedures, the fix involved a relatively simple, automated ride if everything went well, but high-workload critical manual procedures if there was a problem.

Shepard and Mitchell said later that they felt confident about the procedure. They felt the ground had handled the problem well—not burdening them with the details of developing the new procedure, just calling up one that was tested and accurate. Of course, Shepard admitted, ''we didn't have much of a choice. It was either try that or give it up.''28 It will always remain a hypothetical question whether the crew of Apollo 14 could have aborted safely with the new procedures (although the same could be said of the original procedures as well).

Mitchell didn't feel he was tempting fate by typing in such strange mode changes at the last minute. ''I was so darn familiar with the systems that I could play games with them, you know, like a computer hacker.'' He was comfortable with the computer;it was something he'd worked with in many different modes. ''I could do things with the AGS and with the main guidance system that really weren't in there, that weren't even in the checklist.''29 For all its complexity, the human-machine system of the crew and their two computers was flexible, adaptable to new tasks on the spur of the moment. But the new procedures also introduced unwelcome uncertainty.

Relieved to get the procedure into the computer, Shepard said he ''felt like we were home free.'' He realized later that he was being too optimistic—plenty of things could still go wrong.30

Preparing for the PDI, the crew was tense. ''We were trying to be a little light-hearted,'' Mitchell later recalled, ''about the fact that the abort switch problem was really a disaster. I mean, it had us pretty ginchy and gun-shy. God only knows what could happen next.'' He added, ''We suspected, in the back of our minds but didn't know for sure, that when you re-write a system and re-engineer a system . . . you could create side effects.''31

The astronauts went through the checklist for the PDI burn. The RCS engines automatically fired for ullage. Mitchell hit PROceed. Next: DPS ignition.

Shepard: ''Okay. And the Master Arm is off.'' This switch would normally have been on—enabling the PNGS to signal an automatic abort. From now on all such procedures would be manual. About half a minute after ignition, the commander manually throttled it up.

Some of the workload was shifted to Shepard at this point to enable Mitchell to work the computers.32

Mitchell: ''001, Enter. Should have guidance. And you have Command and Throttle.''

At this point Shepard began to feel the procedure was working, as the computer took the craft down the trajectory. ''PNGS was happy with itself. So that gave us a little more confidence that things were going along well.''33

Mitchell: ''You have Command and Thrust. Okay, Houston. The procedure is complete.'' What would have been accomplished by the computer now took tens of keystrokes.

About two and a half minutes after ignition, Shepard keyed in Noun 69, plus 2800, the navigation correction.

They were now into a normal descent. A little bit fast, about ten feet per second, a little bit low. But all within normal bounds. Mitchell reported: ''It looks good, it looks good.''

About four minutes into the burn, Mitchell started looking for the landing radar to lock in. ''Down to 32,000, we should be getting landing radar in very soon.''

No radar. Forty-five seconds later he checked it again. Mitchell fiddled: ''On radar, set the lock ON on radar.'' Then: ''Can't get the radar in.''

The radar was supposed to be coming in (on Apollo 11 it had come in at 37,000 thousand feet;on Apollo 12 at 40,000). ''At 20 thousand feet, that's when were frantically trying to get it to come in,'' Mitchell later recalled, ''because, at 10 thousand feet, there was an automatic (meaning 'mandatory') abort without the landing radar.''34

''We still have ALTITUDE, VELOCITY lights,'' lights that should go out when the radar locked on the surface. Shepard added, nervously, ''I'll bet they know that.''

Houston issued instructions: ''We'd like you to cycle the LANDING RADAR breaker.''

The radar, it turned out, had somehow switched to ''low scale,'' designed for close-in measurement, before the burn even began, which prevented it from locking on the moon at long distances. Recycling the circuit breaker switched it back.35 This was the electrical engineer's oldest and crudest troubleshooting technique, but one frequently effective with balky electronics. Cycle the power: turn it off and turn it back on again. ''Just a body performing the job,'' Mitchell said of himself later, ''searching for a solution to a dilemma, functioning like an extension of the computers at my fingertips.''36

A tense silence followed, as all watched the lights for twenty seconds. Mitchell: ''Come on in... How's it look Houston?'' Mitchell: (To the radar) ''Come on in!... Okay!''

It had popped in. Mitchell went to work: ''VERB 57 ENTER,'' reviewing the radar data. He asked Houston, ''Can we accept?'' Haise: ''Okay. We'd like to accept the radar.''

About 11 minutes after PDI, Apollo 14 passed the high gate. Said Mitchell, ''Okay, there's pitchover.'' He looked out the window and saw his landmark. ''There's Cone Crater.''

He had no trouble recognizing the landing spot. It looked just like the plaster-of-paris model they had used for training. For Shepard, the ''determining factor'' was the ''high fidelity of the simulator visual display'' and the time he spent training with it.37 Tension broke in the thrill of recognition. Mitchell: ''And there it is.'' Shepard: ''Hot damn! Right on the money!''

Unlike Buzz Aldrin, the LMP who had concentrated wholly on the computer displays, Mitchell was looking out, ''stealing glances out the window with virtually every sweep'' of the instruments, he said, backing up Shepard's own senses, giving guidance, suggesting moves.38

Shepard redesignated once, moving the landing point approximately three hundred and fifty feet to the left. As the LM descended, past about one thousand feet, Shepard saw that his redesignated point was no good—the crater there was too large. He entered P66 and flew back to the original spot.39 The commander took over control at about 360 feet altitude, about 2,200 feet short of the target.40 He also flew 2,000 feet down range, because the coordinates of the landing site were in error by 1,800 feet.41

The spacecraft landed on a slight slope, with what Shepard described as ''about a 7-degree right-wing down attitude.'' If he'd come in faster, he thought, the vehicle would have ended up more level.42