Now that Discovery is safely delivered to the Smithsonian, I think I can tell the story of how we nearly lost her in July of 2005, and how well-intentioned, highly motivated, hard-working, smart people can miss the most obvious.

It’s tough to know people who have died. Many of us knew the astronauts on Challenger and Columbia well. We had met with them daily, we had visited in their homes, we knew their families, their children. It is not an easy thing to lose a colleague; especially one who entrusted their safety to you. So don’t question whether we were motivated to prevent another loss.

Discovery was the shuttle return to flight vehicle after the Challenger was lost; two and a half years were spent from January 28, 1986 until Discovery flew in September 1988. Many improvements were made which resulted in a safe space flight.

Discover was the shuttle return to flight vehicle after Columbia was lost; two and half years were spent from February 1, 2003 until Discovery flew in July 2005. Many improvements were made but safety was not assured.

It was not until Discovery again flew in July of 2006 before she flew safely. That counts as the third “return to flight” mission for Discovery.

You see, we dodged a bullet in 2005. One we should have seen coming but didn’t.

After the loss of Columbia, it became very clear that the loss of insulating foam off of the external tank caused damage to the heat shield which led to the loss of the good ship and crew. It was a devastating time. Tens of thousands of experiments, tests, and analyses were performed to discover why foam was lost from the external tank and how to prevent it. One of the most powerful series of test performed used a partial panel, about 2 feet square, of the aluminum skin of the tank covered with insulating foam. This test article was chilled on one side to the cold temperature of the cryogenic hydrogen fuel, and heated on the other side with flames simulating the heat generated by supersonic flight through the lower atmosphere. Adding high velocity airflow to mimic the high-speed flight conditions, virtually all the time the foam stayed on the tank. Only when small defects were embedded in the foam would some of that foam pop out under those strenuous conditions.

All the tests and analyses lead to the conclusion that defects in the application of the foam insulation could cause foam debris to be generated during the early supersonic phase of shuttle flight. We informed the foam technicians at our plant in Michoud Louisiana that they were the cause of the loss of Columbia and then worked them overtime in training with new and exhaustive techniques on how to apply foam with no defects.

Many other safety measures were incorporated during those two and a half years; a new inspection boom with camera was built so that complete inspection could be made before shuttle re-entry; heat shield repair materials were tested and proven to work in the vacuum of space if repairs were needed, plans for safe haven at the International Space Station were formulated so the crew would be safe even if extreme, unrepairable damage were detected. Anybody who followed the shuttle program in those days knows all this.

What you probably don’t know is that a side note in a final briefing before Discovery’s flight pointed out that the large chunk of foam that brought down Columbia could not have been liberated from an internal installation defect. Hmm. After 26 months of work, nobody knew how to address that little statement. Of course we had fixed everything. What else could there be? What else could we do? We were exhausted with study, test, redesign. We decided to fly.

The launch seemed very normal, we had new cameras everywhere including one on the external tank itself. We all breathed a sigh of relieve when Discovery reached orbit seemingly without incident. The members of the Mission Management Team received an early briefing from the video review of no incidents; we boarded our planes from KSC to head back home, that mean Houston for me.

After arriving in Houston, I checked in at Mission control, everything seemed normal. The MER engineers had nothing to report. I headed home after an extremely long and emotional day. It was on the drive home looking forward to dinner and sleep when I got the call.

I think that must have been the worst call of my life. Once earlier I had gotten a call that my child had been in an auto accident and was being taken to the hospital in an ambulance. That was a bad call. This was worse. Think of the worst phone call you have ever gotten. This was the worst for me.

John Muratore, my good friend, fellow flight director, and then the head of the shuttle program Systems Engineering and Integration office informed me in very flat terms that he was in the JSC video lab with head photo interpreter Cindy Evans who had uncovered evidence of a large foam liberation during the critical mach number regime which appeared to have impacted the left wing of Discovery. Just like Columbia.

I was numb.

I made an illegal U-turn in the middle of NASA Road 1 and definitely exceeded the posted speed limit heading back to JSC and the photo lab. Here is one still frame from the video they showed me: A very large piece of foam coming off the tank heading for the wing.

I thought I would not be able to breathe again.

I thought of Eileen and her crew and how we had probably just killed them.

I thought of all the work and time and how wasted it had been.

I had no idea how I could possibly tell the team, the world, what had transpired.

But unlike Columbia, this time the crew had the means to inspect their heat shield. Over the next hours, that inspection revealed no damage. The pictures and engineering data showed no damage. Positive and unambiguous evidence showed the heat shield intact and safe.

Then and only then did I start breathing again.

The next day we told the crew, sent them the video, told the entire NASA senior management, and at the press conference showed the video and made the characterization that this was “unsatisfactory”. A pretty bland word for the way I really felt

So what happened? The video wasn’t really clear whether the foam missed the underside of the orbiter completely or struck such a glancing blow that the impact resulted in no damage.

But that is not the real question to be answered. What really happened to the foam? We wouldn’t find out until December.

God gave us a great gift during the previous year. We had ECO sensor problems. At the time that did not look like a gift. We filled the external tank with cryogenic fuel as a test in May and the Engine Cut-Off Sensors – little wet/dry indicators at the bottom of the tank – malfunctioned; indicated dry when they were wet. We scrubbed that tank from the first flight and sent it back to the factory to analyze sensors.

Now an external tank is a fragile and valuable thing. On a good day the cost estimate for a completed, flight ready tank was $30 million. On a bad day, more than twice that. We never, ever, had shipped a tank back to the factory. We tested them at the launch site, made occasional repairs, took lots of measurements, but never had shipped one back before.

When we shipped it back – did I forget to mention that Hurricane Katrina devastated New Orleans the month after we launched Discovery? Did I forget to mention that the steering mechanism on the trailer the tank was on failed so we sent another tank back to Michoud before the critical one? Months passed and right before Christmas, we got these X-rays from Michoud.

They showed cracks in the foam. Foam where no installation defects existed. It turns out that the thermal cycles associated with filling the tank could crack the foam, especially in areas where there were two or more layers of foam. This never showed up in the partial panel tests; it only showed up when we “tested” a complete tank.

Finally, this explained the Columbia foam loss. And the Discovery foam loss. And it had nothing to do with the improper installation of the foam.

I flew to New Orleans within a few days, and called an all hands meeting where I publicly apologized to the foam technicians. They had not caused the loss of Columbia through poor workmanship. Those guys were reeling from the hurricane’s devastation to their homes and community, and has lived with nearly 3 years of blame. Thin comfort for me to apologize: so late, so little.

We worked feverishly to remove foam on foam wherever we could, minimize it where it could not be eliminated, and the following July we were ready to try again.

Discovery flew on July 4, 2006; no significant foam loss occurred.

I consider that to be the real return to flight for the space shuttle.

So were we stupid? Yes.

Can you learn from our mistake? I hope so.

So when you go to the Smithsonian and see Discovery there, think how lucky you are to see her whole, intact, and with her crews safely on the ground.

You see, this is how I found out that we were never really as smart as we thought we were.

Maybe that is a lesson that applies to you, too.

Share this:

Like this:

LikeLoading...

Related

About waynehale

Wayne Hale is retired from NASA after 32 years. In his career he was the Space Shuttle Program Manager or Deputy for 5 years, a Space Shuttle Flight Director for 40 missions, and is currently a consultant and full time grandpa. He is available for speaking engagements through Special Aerospace Services.

302 Responses to How We Nearly Lost Discovery

Wayne,
I have a few questions. I am certainly not criticizing you or NASA, these are just things I’ve always wondered about.
You said the foam couldn’t be installed on the inside of the tank. You mentioned fuel contamination. What do you mean by that? It’s my understanding that inside the ET were two separate tanks holding LOX and LH. Those tanks couldn’t be insulated independently leaving the main ET uninsulated? Why did we need foam on the outside? Apollo didn’t have external foam insulating the Saturn. Would the ice particles falling off immediately after take off been that dangerous to the TPS?
Also, couldn’t a fine Kevlar screen like material been developed to overlay the foam on the tank so that if some of the foam did break off, it would still be held in place rather then fly off and strike the orbiter?

We received nearly a thousand suggestions during the return to flight period on exactly these same topics. I’m really tired or repeating myself. But here goes. The LOX and LH2 tank skins are the outside of the ET; there is no second wall, so insulation on the outside of the tanks is on the outside and subject to aerodynamic forces and debris strikes. The foam could not be inside the tanks because it would contaminate the propellants. Many ideas of overwrap or inserting elements into the foam were examined, all resulted in more, not less, foam loss, and were extremely heavy (its a big tank). Apollo (Saturn IB and V) did not insulate and big chunks of ice fell off early in launch, clearly would have been damaging to the shuttle tiles.

These have been written up extensively over time; I think a short internet search would lead to more information than I can put in this short reply

I still have a DVD of the press conference; maybe I’ll post the entire video and you can judge if I was misleading or not. We were trying to be completely transparent without becoming overly emotional.

I never felt deceived, and even after reading this I do not feel deceived. But perhaps it is because I am an engineer; this is how we talk. “Unacceptable” is pretty strong language to an engineer. Think of acceptance testing — if your supplier has just shipped you a very expensive controller computer and it has undergone testing and been deemed unacceptable, that means you cannot induct it into your manufacturing facility, you cannot integrate it into any systems or use it for any testing, it is completely worthless to you at present — it is, in short, a very expensive doorstop. “Unacceptable” is strong language.

I remember watching this press conference, actually. The impression I came away with was that this wasn’t over yet, because they could not even contemplate the next flight without resolving this issue — they were stopped, and routine flight had not resumed. “Unacceptable.”

I remember seeing the video of that big piece of foam twirling through the air. Very scary situation.

How do you troubleshoot things like the cutoff sensors? Is everything on the outside of the tank except for the thermocouple or whatever actually touches the fuel? Can a person get into the interior of the tank?

The level sensors are inside the tank and there is access for people to enter the tank (not on the launch pad). The ECO sensor problem was traced to the pass through connectors that carried the wires from inside to outside the tank.

I believe that I am correct, and please correct me if I am wrong, there was a similar problem with pass through connectors in one of the expendable launch vehicle programs and NASA learned a great deal from how they solved the problem.

Wayne,
Few would have gone to the trouble of making a personal apology to those technicians in Louisiana. That was a very honorable gesture. Take it from someone who has given, received, and NOT received such an apology, it is genuinely appreciated and all too rare.

Wayne, thank you so much for telling this story. As a TPS engineer, I know about the foam loss risks all too well. Unfortunately, the story didn’t end here. We continued to lose foam from the external tank all the way through the last mission. It’s worth mentioning that the design requirements for the tank did not allow for ANY foam liberation or debris. The tanks never met this design requirement. And the problem of foam liberation was never completely solved. I attended a TIM (Technical Interchange Meeting) at Michoud in Louisiana after we saw significant losses from the intertank area on STS-127. The spray of debris caught on the ET cameras was disturbing, to say the least. (For those interested: http://news.cnet.com/8301-19514_3-10289011-239.html and http://www.nasaspaceflight.com/2009/07/sts-127-endeavour-rpm-docking-tps-latest/ ) At the meeting, one of the Michoud managers explained the inability to fully understand the material issues and fully control the foam installation process with the following statement, “Foam is foam.” Meaning, the nature of the processes used to make foam (and spraying it on/ allowing it to expand in a mold) cause the material properties to be non-uniform. This makes it impossible to accurately model and predict the behavior of the material. NASA accepted this and thus, accepted the risk associated with foam loss. In my opinion, we dodged more than one foam bullet.

We had thousands of inputs from non-technical people on how to improve the foam; none of those comments – like this one – were useful. Fiber glass would have made the foam much heavier and probably lead to increased debris during ascent. It simply doesn’t work the way you are thinking.

Foam is heavy enough; liberating chunks of fiberglass would be worse. Re: insulation inside the tanks: try suggesting that to somebody who works with the SSMEs and watch the sweat break out on their brow. Foam liberation on the outside: some chance of hitting the vehicle and causing severe damage. Foam liberation on the inside: eek. I think of a chunk of that being ingested by a turbopump.

I remember such suggestions cropping at at the Uplink forum where I used to hang out. We started to get silly; one person suggested a knitted ET cozy, which could be removed at launch by tying the tail to the fixed service structure and allowing the whole thing to unravel. For a lark, I calculated the amount of yarn required, and how large the herd of sheep would have to be to supply it. It was very silly, and completely impractical, of course.

I am an electronic engineer and get a great deal of “help” from non-techs. I understand wanting to help, but it consumes large amounts of time explaining why a solution is not viable. Things such as “well you could use a heatsink made of unobtainium, I read about it on the internet”. He never realized that the unobtainium thing is a spoof. Such a wonder metal does not exist, at least in this universe. The worst help however often comes from engineering techs who know just enough to be dangerous.

Please let me correct your statement. The NSTS 07700 requirement was that no “critical” debris could be liberated. There is no way to prevent all foam losses during ascent. We conclusively proved that very small liberations or those at aerodynamically benign times were not important.

I seem to remember an earlier classified Pentagon mission of Atlantis that came closer than this to ending in tragedy. That was back in 1988, I believe. The crew were not allowed to down-link high res video of the damage to the wing and heat shield due to the nature of that mission. Engineers did not realize how bad the damage was until Atlantis was safely on the ground.

In hindsight, yes, but at the time? It seemed logical that the foam liberation was because of the reasons suggested in the partial panel testing. Don’t beat yourself up too much. You did the right thing when you found out, and not many have the humility to do that.

I watched the launch, and all of the news conferences. I remember being disappointed that, apparently, the root cause of the foam liberation was not quite nailed down, but always knew that NASA would figure it out in the end.

No way to know how gut wrenching this discovery was to you, from your demeanor in the news conferences. It isn’t easy keeping your head, when all about you are losing theirs.

Wayne, I was the contractor project manager for the JSC image analysis lab during this flight. Cindy Evans was our NASA lead and she was the one John Muratore was working with in the imagery (video) lab. I think she was actually the person you mentioned as being the head photo interpreter and not Cindy Conrad.

Hi Wayne: Don’t understand your explanation or any others for the loss of Columbia. When the foam fell off and hit the leading edge of its wing, as I remember it wasn’t traveling at supersonic speed, it was soon after liftoff. I never understood why the problem wasn’t explained to be caused by the hold-down bolt(s) that were known to have broken off. They were heavy chunks of steel not semi-hard insulation. Cheers

This should be required reading for all engineers. Thanks for your insight and humility. Only other comment is that the estimated speed of the foam relative to Columbia at impact was between 416 and 573 mph – obviously enough to punch a hole in the RCC panel, but a little below supersonic (CAIB Report Vol I, p.34).

I liked your story, but as is often the case, there was a side story as well. You mentioned that close inspection revealed “positive and unambiguous evidence showed the heat shield was intact and safe.” I seem to recall that our first look at the Orbiter heat shield in space revealed some interesting observations that had never been seen on the ground and a first-of-a-kind EVA. Perhaps you could tell that story someday…

A gap filler was protruding from between tiles; this was unrelated to ascent debris damage and we can probably still argue about whether the EVA removal of that sliver of plastic was really required. We executed the removal out of an overabundance of caution.

It did result in some pretty awesome NASA TV footage, and what was I’m sure an unforgettable experience for one astronaut. Some would say there’s no point in all that, but you’re up there anyway, and there is engineering value to the exercise, since it affects supersonic flow, so why not? The overabundance of caution didn’t harm anything.

Wayne, thanks for the good read –
I feel sometimes just in regular business, that we all go to the most obvious sign in a problem, and not necessarily look in other directions.

You all did what you thought was right and necessary, and no one can be to blame – All we can do is learn from our mistakes, and hopefully NASA, and the handfull of Commercial Program Partner’s that are currently developing man-rated systems are doing the same thing.

Thanks Wayne…….another potential “black swan event” averted only by luck. Through the terrible events and close calls experienced during the space shuttle program, I have learned not to “normalize deviations” and to embrace and encourage messengers with safety concerns.

During the early days of the program I was involved with ground systems development. Our highest priority projects were characterized as having the potential for “loss of vehicle and crew” if not accomplished. At the time, none of us really believed that would ever happen because of the rigor that was applied to anticipating the unexpected, reducing vulnerabilities, targeting potential consequences, and evaluating possible impacts. It’s a shame that this rigor didn’t prevail over the life of the program.

As we move forward to the next government and commercial human spaceflight programs, program managers, engineers, and technicians should be required to take the sad walk through the Columbia Room on the 16th floor of the Vehicle Assembly Building at KSC where the recovered pieces of Columbia are displayed. It is a grim reminder of the devastating consequences of poor decision making and failed management.

Thanks for the story, Wayne. Exploration is always risky. Hopefully, we’ve learned something that may be of use in the future, if only to be not too quick to think we know all the answers…or even if we’ve asked the right questions. Hopefully America will soon resume human spaceflight and press forward beyond LEO! No doubt, there will be rough times ahead, but we will maintain our resolve to head Ad Astra!

This story, combined with Victoria’s TPS statements above (“Foam is foam”) should be more than enough to convince any reasonable person that any future manned spacecraft should be placed on TOP of the launch vehicle, not slung on the side of it in the potential debris path.

I am happy to hear your insights into return to flight and thank you for shedding light on this harrowing time in our space program’s history. Recognizing that we cannot ignore those “nagging one-liners” that speak truths inconsistent with the conclusion we have come to believe is a lesson that must be learned and [sadly] re-learned.I had the pleasure of meeting @OV105GirlEngr yesterday. Her comment made me thing again of a sad coda to the end of the Shuttle era: the number of talented, experienced, and valuable engineers like her and other workers who are now out of work.

Will we ever learn that by “booming and busting” our aerospace workforce, we discard thousands of combined years of experience in each cycle as people who must feed themselves and their families depart the industry in search of steady careers?

Just as you have known and mourn those who died in the pursuit of our space program, surely you have suffered the less intense–but more frequent–loss of colleagues driven away from their one-time “dream career” by economic forces. I’d love to hear your observations on that and any “lessons learned” you have on how we can improve the situation.

Sir, perhaps you simplified the situation for your story, but It seems a little tough to have “informed… the foam technicians.. that they were the cause of the loss of Columbia” when, as with most aviation or space disasters, there were many factors contributing to the outcome. One of these is the design itself, which exposes relatively fragile foam to a supersonic environment. Another one, which bothers me no end, is the (reported) failure of NASA management to pursue DoD imaging of the shuttle on orbit, as requested by NASA personnel. Had such imaging confirmed what these personnel suspected, the outcome might have been the same; but possibly not. As with Apollo 13, a lifesaving improvisation might (might) have been devised.

Man, I really sounded like a jerk. I better backtrack here and say, first of all it’s amazing that a former NASA manager of your caliber is writing a blog in the first place–thank you–and secondly, I did not intend to demean your actions wrt to the Michoud techs.

To apologize to those guys like that… you had to have made their lifetimes.

Actually, some of it is public knowledge. When I returned home from vacation in Myrtle Beach in 2003, a newspaper story about how NASA engineers were concerned enough by the ascent imagery they had that they began going through their own “back channels” to obtain better on-orbit imagery. I believe that it was a mission manager (who shall remain nameless because no purpose would be served by naming names) who scuttled the operation by saying “I don’t believe that there is anything we can do.”
This also showed up in the numerous TV news programs that I recorded in the wake of the tragedy.

Some of that is reported in the book “Comm Check”, which is a very emotional read; it started out as a fairly dry overview of the CAIB findings but the author found it turned into an exploration more of the people involved. It’s really worth reading. Nobody was being bad or callous; they were all doing what seemed to be the right thing, and that’s the devil of it. I highly recommend the book, though as an outsider, I can’t attest to its accuracy. (I’ve never heard anyone say it was inaccurate, though.)

A lot of armchair space fans figured the thermal cycles were going to make that tank a disaster. The problem is people do not get jobs based on imagination or intuition but single words like SolidWorks.

Armchair space fans figure out a lot of things; sometimes, they’re actually things that are true, but not always. Not to be too harsh on my fellow armchair space fans, but even a broken clock is right twice a day, so predicting this doesn’t mean they should have been listened to about it.

I can’t believe that you, with your degrees, cannot grasp what his point here is. He’s trying to infer that mechanical vibration can be a contributing factor in the shedding of foam. It’s simple, really…how many flex cycles at a given amount of deflection at a given frequency of flexing does it take to cause the foam to (a) lose its ability to adhere to the tank surface, and (b) begin to crack along stress-induced fault lines?

During the post Columbia days we received many communications from the public. Several people proposed that it was harmonic vibrations between the celestial spheres which caused the foam to come off. A rather large contingent wrote in to say that harmonic electronic radiation was the cause of the foam coming off. So when I ask what the commenter meant by harmonic, it was because of those rather unusual communications I remember getting. Finally, the affect of vibration and the potential to excite natural resonance frequencies in the structure and insulation of the tank were well recognized and thoroughly tested. I appreciate that I may not have been clear earlier in my responses.

I was reasonably certain that foam strikes had occurred long prior to losing Columbia, and that NASA engineers had come to view them as inevitable and (relatively) harmless. If this is the case, hadn’t we “Nearly Lost” all the other shuttles that were hit by foam?

“Comm Check” mentioned the one Atlantis had a couple of missions earlier, where foam struck extremely close to a critical electronic box on one of the SRBs. A few inches to one side, and it would’ve killed the crew by destroying the SRB’s control. I seem to recall from the book that it was ultimately more-or-less written off as a freak accident.

“We probably had a lot of near misses before Columbia and were just unaware.”

As I understand it, the temperature conditions did for STS-51L were actually experienced on many other flights previous to that one (which was the reason they thought it would be all right.) And later research showed that it wasn’t a one-time thing, that the O-ring response to temperature was consistent. So there were plenty of near misses for the Challenger-type failure before STS-51L. Indeed, if there’d been another serious failure, they’d probably go back and find that there were near misses for *that* failure mode.

What this means is not that NASA is incompetent. What it means is that spaceflight is enormously complicated, and you generally don’t see that something is wrong until it blows up your rocket, because there is so *much* that you need to look at. It’s easy to say “oh, well OBVIOUSLY we should have been looking at (thing)” afterwards, but before the accident you were looking at a lengthy checklist of other things–any one of which *would* blow up your rocket if you didn’t watch them.

Nope, 51-L was by far the lowest temperature launch. There were other launches at cool temperatures but not that cold. And you are right, post accident data review did demonstrate a correlation between cooler temperatures and increased o-ring damage. There are several excellent books out on the Challenger accident, I think the best is by Dr. Diane Vaughn, “The Challenger Launch Decision”. If you want to know more, I highly recommend that book.

BTW, we were entirely incompetent for both the Challenger and Columbia accidents. Both could have been prevented; the data was in front of us and we were not paying attention. The lesson, for all high risk/high reliability operations is to pay attention; do not be distracted.

Thank you for sharing your thoughts and feelings.
It’s refreshing to hear such deep emotions and humanity from someone involved in such cool and calculated endeavors.
Learn from the mistakes of others. You can’t live long enough to make them all yourself.

As a short follow-up to my earlier comment, the bolts were supposed to be caught in baskets to avoid hitting the shuttle. As I recall at least one was found on the launch pad. Any one else remember this? Cheers

Agreed, that phrase jumped out at me as well. If indeed the foam technicians were ‘informed that they were the cause of the loss of Columbia’, then NASA’s problems continued to persist even then. Initial design/performance standards, internal fabrication methodology guides, Unit QA checks, etc *all* would have appeared to have failed. Foam technicians, various engineers of different specialties, prep personnel, etc do not exist or perform in a vacuum .

If the foam technicians were the cause of the loss of Columbia, then they have the close company of the organization as a whole. Seal teams live and die as one. NASA should be prepared to do no less.

This is such a strange comment that I don’t think you really understood what I was trying to say. We were wrong in what we told the workforce at MAF; that was my point. I don’t really know how to compare this to a US Navy Seal team.

I noticed the sharpness of the way you initially addressed the MAF folks. At first I figured that it was necessary to impress on them the importance of the work. But I’ve read all your columns for years, and have such respect for your engineering, leadership, and writing, that I withheld judgement in my reading until the end.
Sure enough, in the end it seemed to me that your retelling of this story, with the initial harshness to MAF, was exactly calibrated to show all of us that humility is a required. I for one sure appreciate that lesson. The way I read your article, the story was inspiring to acknowledge the chances of being wrong while in a difficult analysis and decision process, and to acknowledge publicly the mistake to those effected after the fact.
As I said: inspiring. Thanks

Perhaps I was a little too brief in my writeup. Nobody that I know (certainly not me) went to MAF and downdressed any of the workers. What happened was that the conclusion was reached in engineering and management meetings and the word filtered out to the workers that poor workmanship was the proximate cause of the loss of Columbia – as if there wasn’t enough blame to go around in many other areas. I really regret the erroneous conclusion, the impact it made on the workers, and the way the whole scenario played out at MAF. The people there were very hard working, dedicated, and proud of their involvement with America’s space program, many of them second or third generation workers at that location. Now, of course, they have all been laid off and the MAF plant is virtually a ghost town with very limited work going on there for other NASA or commercial projects.

Thanks Wayne. The only way for future generations to learn is to hear the lessons of earlier generations – both the good and the bad. And overall we get plenty of the “good” stories, but not enough of the “bad”. You’ve been doing a good job of providing both. Keep it up.

Wayne…very well put. Each and every one of us needs to heed your reminder of avoiding intellectual arrogance. Everyone is capable of error, regardless of their position. Thank you for sharing this important story.

Thank you for an amazing post. The ability to work with the stress of the job must have been brutal. You did the best you could given the information at the time. Hindsight is both a good and bad thing – it makes you second guess your actions. Despite the issues, you got a lot of brave people back safely from one of the harshest and most dangerous places imaginable. Not an easy task at the best of times. Good job.

Even though application technique wound up not being the cause of the Columbia accident, I would think that the extra diligence, training, inspection and analysis of foam application after Columbia contributed greatly to the safety of the Shuttle. As did the redesigns of foam around the ramps, bipods and bellows. As did the amazingly thorough and time consuming on-orbit inspections of the TPS, very precious on-orbit time that was dedicated to a striving for safety. And the very thorough and skillful analysis of the inspection data by ground teams. So skillful that on one mission, while the Shuttle was still in orbit, the laser data from a damaged tile was downloaded to computers and machines on Earth which were able to recreate an exact duplicate of the tile including damage, so that it could be run through an arc jet to simulate re-entry and create even more data for the ground teams to analyze.

Even all this did not guarantee that there wasn’t the possibility of severe TPS damage due to foam loss. However our choice was to either stop flying after Columbia, or keep flying and keep improving. I for one am glad that we chose the latter because the post Columbia missions accomplished many things and gave us tremendous additional knowledge about how to overcome the incredible challenges and difficulties of space travel.

Hi Wayne
Thank you for letting us see inside your world during this terrible event. As a former KSC worker (Bay 3 was my Bay) I honestly do not remember any blame being placed on the MAF Techs. Of course everyone (including me) was in shock, coming back to find Bay 1 empty –you could have told me the sky was green and I wouldn’t have noticed. I helped to bring Columbia home which was a privilege and I cried when Discovery (our come back kid) returned us back to space. Even with the risks from potential foam strikes, I think the shuttles should have kept flying until Constellation / Orion was ready to go. People ask me if I would go up, and my answer remains the same –“YES, any day of the week and twice on Sunday!!” I know the hard work and dedication put into getting these birds ready for a mission –No one did it better : )

I love this comment. Having supported five Hubble servicing missions, I have come to know many KSC colleagues and found them all to be incredibly dedicated and capable people. I miss the process of seeing them every few years on our visit to KSC. I wish them all well where ever they are.
PS. I am heading out this morning to go volunteer at the Udvar-Hazy and look forward to seeing Discovery with Mr. Hale’s blog fresh in my mind.

I too agree that the shuttles should still be flying, I went to the last shuttle launch of Atlantis and I will never forget the empty feeling when she disappeared behind the clouds because I would never again see such a beautiful machine launch into space again! I envy EVERYONE that ever got to work at KSC that has to be the most amazing job in the world

“All the tests and analyses lead to the inexorable conclusion that defects in the application of the foam insulation could cause foam debris […] We informed the foam technicians at our plant in Michoud Louisiana that they were the cause of the loss of Columbia ”

That seems rather harsh, considering that it’s an argument based on ignorance. Even if one particular possible cause was identified, you couldn’t eliminate every other possible cause.

This leads to how NASA examines failures. Typically a ‘fishbone’ is created to look at all the possible causes, however remote. The question I have is whether the “thermal cycling causing foam cracking” was even on the fishbone associated with Columbia?

If so, how was it removed from consideration? If not, how can we change our processes to include things we’ve really not thought of?

Also, I wholeheartedly agree with your line on realizing that we’re not as smart as we think we are. Too often we convince ourselves otherwise.

Wayne, congratulations on the success of your fix. I’ve had a lot of experience in apprising senior management of the impact of reliability problems & the effectiveness of the fixes to them. I was taught that the words you use are dramatically important, dramatically impressionable & that you had to choose them very carefully. How did you come to use the word “foam?” To the layman, “foam” is soft, light, airy… how could it hurt anyone. How could we be spending time & money & worrying about human life, because of “foam?”

Foam is what it is called. It is light and airy. Part of the tragedy of Columbia is that none of the managers (including me) could imagine how such light and airy stuff could cause catestrophic damage. If light and airy stuff hits you at 750 mph, it will do damage.

If you rode a motorcycle you would have understood intimately the relationship between velocity, mass, and damage.
In the days after Columbia’s loss, someone at NASA went to press saying that the foam had the same impact as a foam beer cooler blown out of the back of a pickup truck doing 55 mph. How they chose that metaphor is beyond me, because I have had that happen while riding and it’s only by the grace of God that I’m here!
I’d love to be able to say that they woke a sleeping giant, but a mental midget would be closer to the truth. LOL!

Being in the business you’re currently in, you know all about how disasters are rarely caused by only one thing, rather, they are the result of an accumulation of things that appear insignificant at the time. It’s only when you peel off enough layers of the “onion” to expose the raw center that you get into trouble.
The recent centennial of the sinking of the Titanic is a good example. It’s been analyzed using modern techniques from the engineering as well as the human performance perspectives.
Maybe because you lived it is why you cannot see it. I’m a year younger than you, and have been observing how NASA works from a distance for years.
The CAIB report seemed to cover this issue in detail. Nevertheless, schedule pressures resulted in the decision to launch STS-133 on December 17th, 2010, after finding cracked stringers on the ET when foam came off during a tanking test. Just like the ECO sensor issue, this was also a Godsend. It revealed an issue that would have remained hidden until the SRBs ignited and pushed the LH2 tank into the LOX tank because 94 out of 108 stringers were defective.

As a taxpayer and not part of the space program, I question how appropriate it is that these untold stories are untold in the first place. What is the rationale for NASA having hushed this up all this time? Isn’t that what we usually call a cover-up? When things go wrong, NASA wants citizens to understand that it’s a risky business. But by glossing over and just plain hiding dangers, it encourages the opposite belief. Did you decide on your own that it was all right to tell the whole truth, finally? Will NASA management be mad at you? Lots of obvious and frustrating questions are raised by these Now It Can Be Told sorts of reports.

Wow. The story was told; when I said that it was time for me to tell my story, it was my personal reaction, my personal feelings. The facts, the events, the technical conclusions were never hushed up, we tried to be completely transparent. NASA management is not mad at me. As far as I know.

Those ships were lost and their crews killed by design errors made decades ago: the decision not to develop a liquid-fueled flyback booster, the decision to use a winged gliding orbiter instead of a vertical lander, the decision to continue with an external hydrogen fuel tank rather than an internal tank even after vehicle mission requirements changed… in short, the decision to continue the program when NASA management should have told Congress that they had not been appropriated enough money to do it properly. Now, finally, after 40 years, private companies are working to do it right.

–*Only when small defects were embedded in the foam would some of that foam pop out under those strenuous conditions.

All the tests and analyses lead to the inexorable conclusion that defects in the application of the foam insulation could cause foam debris to be generated during the early supersonic phase of shuttle flight. We informed the foam technicians at our plant in Michoud Louisiana that they were the cause of the loss of Columbia and then worked them overtime in training with new and exhaustive techniques on how to apply foam with no defects.*–

The whole piece is disconcerting, and, one can only suspect, emblematic of the problems that have plagued NASA for years. The myriad “small defects … embedded” in Hale’s post, the typos, misspellings, ungrammatical constructs, disagreements in tense, incorrectly assigned possessive adjectives, and reliance on jargon, make me wonder whether Hale’s sloppiness and aura of literary ineptitude are indicative of a general failure to perceive problems and communicate solutions.

For instance: The “conclusion” in Hale’s paragraph above might have been inescapable, but it certainly wasn’t “inexorable”.

Elsewise, was someone really so heartless and stupid as to tell the foam technicians that they caused the loss of Columbia? And how could anyone even think that, if the foamers were then to be trained with “new” techniques designed to mitigate the problem? Who trained them with the “old” techniques? Wasn’t it really that person’s fault? What about all the other people who watched foam flying off earlier flights and brushed it off as insignificant? Did no one consider the entire life cycle of the foam after it was applied to the tank?

And why do we only hear this story lo these many years later?

As a kid growing up in the sixties, I was enthralled with NASA’s exploits, but in the last decades have come to disdain it as one more inept, corrupt government bureaucracy. Hale does nothing to dispel that notion.

Your sarcasm is further indication that you have long been out of your league. Your “writing capability” is a window into your thinking capability. As far as I can tell, you haven’t been employed as an engineer for decades. You were an engineer who was kicked upstairs, promoted to mission manager, a person whose job it was to decipher and correlate large amounts of data, and then communicate the relevant parts to the appropriate people. And yet you can’t write a simple blog post that doesn’t call into question your ability to intelligently communicate. If you think the data lead to an “inexorable” conclusion, then you don’t know what “inexorable” means, and if you don’t know what “inexorable” means, then it’s a bit tricky to assert that you know just where the data was leading.

Words mean things, and if your job is communication, they you better be able to use words properly. Your post unveiling the heretofore untold story of “Discover” 2005 does not support the idea that you were, or are, an effective communicator.

That is probably why I’m retired and writing my personal blog on my own personal time. Just consider me another old codger writing letters to the editor of the local newspaper because I have nothing else to do.

I have always felt inadequate as a communicator, and you are exactly right that I have not been a practicing engineer for many years.

The heretofore untold story about Discovery (not Discover) is really the heretofore untold story of my personal feelings and experience. The actual events were factually reported at the time.

In my opinion the measure of effective communication is not whether someone “incorrectly assigned possessive adjectives”. The true measure of effective communication is when the listener understands the ideas being presented by the speaker or writer.

One reason why so many people have enjoyed reading Wayne’s blogs over the years is because he has the rare ability to take extremely technical issues and present them in a way that people understand. Not very many people can do that. And many who do don’t have a lot of depth to their knowledge. Wayne has both good communication skills and a depth of knowledge which is a rare combination. He is also an effective editor who knows how to distill an overwhelming amount of information into a few paragraphs which convey the essence of a story. And he is a great story teller. This combined with his unique experience and insight into the space program makes for fascinating reading. If there are occasional “disagreements in tense” we somehow make do and have no problem understanding the message being conveyed. Perhaps that’s because we realize that this is an informal session, a personal blog where ideas are king, and not a book or magazine article where grammar is a bigger concern, and where the author must spend a lot more time checking and re-checking their grammar and then have an editor proof it as a final check. We can read articles all day long that have this type of grammatical perfection but which don’t convey one tenth of the information that Wayne somehow fits into each of his posts. That’s why people keep coming back, and why we look forward to each new blog post from Wayne. We appreciate them.

I’m an engineer, and my most important skill is communication. If you can’t articulate *why* a particular requirement can’t be met–or understand the reason the requirement exists in the first place–then all you get is twelve people arguing for two hours, with no solution.

Sarcasm is an indicator of waynehale being out of his league? Writing capability a window into…??? Did that nonsense make you feel better about yourself? It would seem everyone else reading this thread understood waynehale… except you… Therein lies the heretofore(lol) inexorable conclusion…

First, mikesoja is not a proper name because there are no capital letters.
Second, who uses a word like “elsewise” in blog comments, except just to be pretentious?
I am a proofreader for a medical printing company, and you would be astounded at how many respected medical doctors can barely communicate on paper. I have grown to understand that these doctors have devoted their lives to saving/improving other people’s lives and maybe didn’t have the time left over to study english to a point of OCD perfection. Mr. Hale’s writing, though admittedly a little rough, is more than sufficient for personal recollections on a blog, and certainly not to be assumed as some official NASA statement. His kind and humble responses to your criticism are a good indicator of his character.

That is an excellent response. I think Mr. Hale did a fine job and at least he is honest and forthright. We don’t really need engineers like “mikesoja,” there are plenty of others with a better character and personality; the kind who you want to communicate with. This holier than thou “mikesoja” would be a primary candidate for causing a disaster and losing a lot of lives. I think he would be too vigilant about attacking his crew members and losing his SA over it. I pray that he never is responsible for any safety engineering or management. Thank you Wayne!

It is nice to see how many NASA and highly trained science and technical people follow WUWT. So much for deniers being anti-science!

Wayne, CJCold’s comment is valid. If something resonates at its natural frequency you get maximum energy transfer, making foam loss or component failure a much higher probability. I really enjoyed your article but was quite surprised at your rebuff of his cryptic comment.

Wayne
I have an MS in AeroSp Engr (PSU ’71) and wanted to work for NASA, bad time to graduate. I worked 30 years in industrial instrumentation industry, last 5 with ENDEVCO for shock & vibration sensors. From the videos I’ve seen, the source of the foam seemed to be concentrated in a specific area of the EFT. If that is true, vibration in that area, due to tank geometry, could be a factor in the foam ablation. Was that even considered?

It’s Watts. Anthony Watts is the author, so it is a play on his name. He mainly blogs about weather and global warming, though he does not limit himself solely to that topic.

He’s the largest “skeptic” blog out there in regards to global warming. There are quite a few NASA scientists and engineers (current and former) that read it on a regular basis. He pointed us your post.

Sir, I got knots in my stomach reading your story. And yes, I have figured out a time or two that I was not as smart as I thought I was; thankfully with no lives on the line… Thanks for the article!

If it’s not too presumptuous, I have a bit of a somewhat related side question about something that’s always bothered me: If only you all had certain knowledge that the left wing of Columbia was damaged beyond repair and could not be landed safely, was there anything in your opinion that could have been done to get the astronauts back to Earth other than trusting to luck? I’m guessing probably not? And that’s why you described the phone call from Jim Muratore as the worst phone call ever in your life, and that’s why you thought at first you all had probably just killed Eileen and her crew? (But since STS-114 was a mission to the ISS, didn’t they at least have the option of staying up there until a rescue mission could be mounted?)

Wayne, If NASA had known the damage to columbia’s wing was as bad as it was could they have stayed in the shuttle until another could be readied to go up and rescue them? or could they have moved to the ISS’s orbit and stayed there until a rescue? and what would have happened to the space shuttle if a rescue had to be performed?
Thank you

The short answer is No. Even a rush job for another launch wouldn’t be quick enough even if you risked another orbiter. Also, the orbiter did not have enough fuel to go to the ISS. The Hubble mission had a rescue on the pad just in case for those very reasons.

Thank you. This should be required reading for just about everyone. It is a storng reminder the kind needed as subtle ones are usless, that we humans are only as cleaver and smart as we permit ourselves to be. The only assumption any of us are allowed is we are not as smart or cleaver or as well educated as we think we are. And yes if someting can go wrong it will and that something is anything.

Wayne,
If you remember, I was at Michoud during the tank re-work after the disaster. Also, if you remember I wrote a report documenting many issues with the external tank; among them cryo-pumping, lack of proper testing, irregularities in the hand spray process and several others. I also expressed my deep concern with returning to flight with these open issues. None of the serious issues were ever addressed or fixed. The answer to my concern with the open issues was that the Shuttle was going to be retired in a few years and the fixes to these major problems would take too long to solve. My report somehow (and I had nothing to do with it) wound up on the front page of the New York Times on August 5, 2005. Following this, I was threatened by Lockheed and hounded by the press until the launch actually resulted in significant foam loss. After that, the ridicule subsided although many still believe that I gave the story to the news media and pretty much let me know that I had somehow betrayed NASA.

Like you, Shuttle safety was always parmount in my thoughts and deeds. However, the ridicule that I experienced after I tried to bring light serious issues with the return to flight configuration of the external tank exposed a truly dark side to my NASA experience.

Thank you for this very touching and revealing story. Murphy is always working. In a system as complex and dynamic as a shuttle, Murphy can have a field day. But this time you beat him. …a good recent book on engineering failures is Henry Petroski’s “To Forgive Design: Understanding Failure” which talks about our propensity to assume too much, misread signals, get complacent, etc. He discusses bridge failures but also the Challenger and Columbia losses. And he notes how failure may come from a poor design, poor operation, poor inspection or repair; or from the interaction, where a bridge (Silver Bridge) failed from stress corrosion at an eyebar link that by design could not be inspected: the designer failed to understand the possibility of such events and so failed to provide for ways to spot and prevent them. In the Discovery foam tank, it seems a similar oversight occurred. People simply didn’t understand how foam behaved through the full range of conditions that obtained during its application, during tank testing and fueling, during flight. Luckily they caught this in time. But they didn’t catch Murphy.

“What you probably don’t know is that a side note in a final briefing before Discovery’s flight pointed out that the large chunk of foam that brought down Columbia could not have been liberated from an internal installation defect. Hmm. After 26 months of work, nobody knew how to address that little statement. Of course we had fixed everything. What else could there be? What else could we do? We were exhausted with study, test, redesign. We decided to fly.”

I’m really not sure how to respond to this without coming off as a jerk, but it’s a very disturbing revelation. This is why Columbia was lost, and Challenger before her.

For two years we heard nothing but how nothing like that would ever be allowed to happen again, and then it does on the very next flight…

What happened again was that NASA’s bias to fly overruled the knowledge that there had been a critical mission failure, that you did not know the cause of, and therefore could not have corrected. You had NOT “fixed everything”. And you decided to fly?! What else could you do? I’m an ASQ Certified Quality Engineer, and I’m sure there were many of my colleagues at NASA at the time that could have told you what to do. Do a root cause analysis of the failure, and then fix the root cause. If the cause you fixed couldn’t result in the failure mode observed, you aren’t done yet.

I don’t mean to belittle the great work that so many did at NASA. I worked in quality on a number of pieces of the shuttle myself. But a bias in favor of flying is the organizational root of the fatal failures.

To Geoffrey below: as James Cameron pointed out a year after the loss of Columbia, “While safety is very important and must be considered at all times, in exploration safety is not actually the most important thing. In exploration, the most important thing is to go.”

You cannot fix everything or even understand everything. At some point, you have to either light the candle or mothball it knowing you will never fly again, because neither time nor money or resources are infinite. You do everything you can to understand and fix as much as possible, but eventually you have to go or there is no point in the exercise.

With respect, a very well told tale. My Kudos’s to all those that undertook the extreme challenges of Human Space Flight. Quite a challenge indeed it is, and in total, a very risky job well done by all.

I have a question about the Solid Rocket Boosters, my understanding is that the original plan was to make them as “one-piece” units on the Gulf Coast and then transport them to Florida via barge. This would have eliminated the need for “O-Rings” totally. However, a US Congress person from Utah intervened and forced the SRBs to be made in pieces in Utah and then transported and reassembled with the requisite “O-Rings” in Florida.

Any insights ? Perhaps this is an “Urban Legend” that I am recalling incorrectly ?

Cheers, Kevin. Thank you for your very professional efforts towards advancing our capabilities…………

So NASA never released any RFQs (Request For Quotation) for a SRB manufacturing factory on the Gulf Coast of the US in the mid 1970’s ?

I was quite an active FAN of NASA back in those days, and I sure remember descriptions of how the SRB’s were going to be manufactured right there close to the launch site and it
was going to be SO easy.

Rather than get in a difficult discussion, I’ll just take your word for it.

“They showed cracks in the foam. Foam where no defects existed. It turns out that the thermal cycles associated with filling the tank could crack the foam, especially in areas where there were two or more layers of foam.”

It is always the “little” things. It is never the “big” things that get you. I have been a maintenance technician on everything from aircraft to air tools over the last 30 years. It never fails — it is the so-called “little” things that bite you.

Excellent account. Thank you.

PS: It would be interesting to know what kind of foam it was — chemical composition, method of application, temp ranges for application, etc.

With respect, a very well told tale. My Kudos’s to all those that undertook the extreme challenges of Human Space Flight. Quite a challenge indeed it is, and in summary; a very risky task well done by all.

I do have a question about the Solid Rocket Boosters, my understanding is that the original plan was to make them as “one-piece” units on the Gulf Coast and then transport them to Florida via barge. This would have eliminated the need for “O-Rings” totally. However, a US Congress person from Utah intervened and forced the SRBs to be made in pieces in Utah then transported by railroad to Florida for reassembly with the requisite “O-Rings”.

Any insights that you care to share? Perhaps this is an “Urban Legend” that I am recalling incorrectly ?

Cheers, Kevin.

Thank you for your very professional efforts towards advancing our capabilities…………

You might be recalling the filament-wound SRB project. Construction of the plant began near Iuka, MS and if I recall correctly, it was shutdown just as plant construction was complete. The plant was built on an inlet of Pickwick Lake which would have had access to either the Tennessee River or the Tom Bigbee Waterway for barge transportation. I believe the filament-wound SRBs were in one piece instead of segments and were lighter than the conventional. SRBs to provide more payload to be launched from Vandenberg.

The story of filament wound SRM cases to be used for polar launches from Vandenberg and the story about the advanced SRM factory at Iuka are different. I’d probably need some help to de-conflate those two stories.

First and foremost, thank you for your service to the greatest adventure. I hope that my generation can give rise to similarly driven and talented people to continue your generations’ work. I often regret my decision to pursue life in the private sector, instead of becoming an aeronautics engineer (or, as the 8 year-old-me would put it, an astronaut – because all engineers naturally get to go to space).

I do have a question, though perhaps it’s too broad and complex to adequately address here. If the opportunity had existed to eliminate the layered foam, in favor of another solution, what might that solution have looked like? In other words, did you and other engineers ever identify a solution (even an infeasible one) that you felt was significantly better than simply minimizing the amount of layered foam?

I understood that to be the case. I suppose what I truly meant requires further explanation. Engineers of all breeds are often forced to choose points on the ‘project management triangle’ (http://en.wikipedia.org/wiki/Project_management_triangle), balancing solutions based on their impact on the project’s budget, schedule, and quality constraints.

So, if budget and schedule had not been constraints, and the emphasis was only on ‘getting it perfect, at any cost and delay’, would the implemented solution have been any different? I have no axe to grind, I am just curious from a technical angle if there were better solutions that didn’t work in the real world due to the above constraints.

This all points to why we should have kept the program going. The technical experience alone is worth the cost. I am a technical person and firre reinforcement is a viable approach. Re. foarm restructuring, Carbon Fibers placed as a compressive element over the foam to reduce shear forces during tank filling. I am not trying to kick a dead horse, but in one breath to admit that everyone is not as smart as they thought they were and in the next to tell someone that they are non technical and their input is useless and that things don’t work like they think is just more of not know what it is you are not knowing.

Wayne, the earlier shuttles had a white tank, we were told the white was paint and that it was decided that the paint was unnecessary so later flights had the familiar orange tank. In hind sight do you think there was an unknown need for the “paint” to be present? Could you clarify about this “paint” as, at the time, that is the story we were told. I would like your more personal account of the issue please, if you remember after all this time.

We did many tests on with and without the paint. The first two tanks were painted white because we thought it necessary to help keep the tank cooler (reflect sunlight). That turned out to be unnecessary. The paint on that big tank added several hundred pounds of weight. It had no affect on the strength of the foam, but it actually contributed to debris (due to the added weight). Painting the tanks again would have made the problem worse.

So it really was just plain old paint.
“Following the Feb. 1, 2003 loss of Columbia during reentry, which investigators found was due to heat shield damage caused by tank foam insulation debris at launch, NASA revisited the white paint approach, as well as a myriad other schemes to reduce foam shedding.”
Not even two part paint? Just a coat of exterior emulsion? Anyway, it still looked better with the paint to me. Thanks for your answer and good luck.

I always appreciate attempts to improve my grammar and vocabulary.
But how can you make a judgement about my feelings? I stand by my comment, not at all fanciful, that was the worst phone call I ever received.

Wayne, I’m not disputing that you believe that phone call was the worst you’d ever received. But if I think about the worst phone call I’ve ever received, which your statement asks me to do, then mine trumps yours. And I’m sure plenty of other readers have received worse news by phone than being told there’s a possibility of a disaster. Being told a disaster has occurred, for starters.

And my comment about communication was a shot at mikesoja . . . I’m always amused when a correcter needs correcting. I had no trouble following your story – typos, poor grammar and all.

I wrote this particular post fairly late at night, after a long and busy day, with less than my usual care. I apologize to all the readers for the typos and grammar errors. I did a spell check but you know that doesn’t always catch everything. I have learned my lesson and will attempt to do a better proofreading and editing job before posting again.

Mathematics is the lingua franca of Engineering… Comments about grammar and communication just serve to show the poster must have been a student of some liberal art. In my opinion, if the Non-linear Partial Differentials are correct? One grunt or two will suffice…

Did anyone, in or out of NASA, ever suggest a return to painting the foam as a way to reduce aerodynamic stress on it? Paint couldn’t have offset all of the weight savings from the lighter alloy with lithium.

How smooth/rough was the unpainted foam surface? A paint with a high film strength, formulated to bond strongly to the foam surface, should have been able to improve the integrity of the foam’s surface while also reducing drag.

Also, we came close to losing Atlantis on flight STS-27, the second flight after the Challenger disaster.

It was a DOD classified mission and the astronauts (from what I read were not allowed to transmit detailed photos down to NASA because it might be intercepted by Russia, etc. and thus military secrets might be discovered). I remember the pictures of Atlantis after landing and the streaks of tile damage on the very forward part of the starboard side forward of the wing. I believe one tile which had been missing showed an almost burn through of the aluminum structure of Atlantis.

We were lucky then too. And again mission and crew safety were put second to DOD “National Security.”

Regarding harmonics and foam behavior at supersonic speeds, could vortex shedding have produced cavitation and literally sucked chunks of foam off the ET? I am no engineer but Petroski’s discussion of the collapse of the Tacoma Narrows bridge –due to high winds generating vortices off the bridge deck, and these pressure changes driving the deck to its resonant frequency– make me wonder. Also, if there was a danger zone for foam from the ET hitting the shuttle, could the foam in that zone have been selectively reinforced with fiberglass or other means? No need to load the whole surface of the tank with it. I’m sure this and many other better ideas were considered. Again, thanks for your service, for your blog, and your courtesy and patience.

I followed this article through wattsupwiththat posting a reference to it. I’m glad I did. I worked at MAF until being displaced by Katrina and never knew the foam application process was eventually exonerated. I’ve been thinking since 2005 (and earlier) that it was workmanship, even though I felt the NASA design was a contributing cause. Even though this story was posted only now, please know that it gives me back some pride I lost. I’m so sad that our astronauts were riding on hardware that wasn’t properly thermal cycle (design) proof-tested. They deserve better.

To me, THIS is the lesson learned.

How does this relate to WUWT? Maintain your skepticism. Question everything. Show me the data. Never assume. Look at the RAW data. Verify axes values, starts/ends, units, everything. Be a scientists, not a member of a “church” of scientific thought.

A life well lived, a happy life, is one in which you can look back and honestly say “I made good decisions.” You sir, lived a happy life in my opinion. Good decisions are not necessarily ‘right’ decisions. They are good by virture of the fact that they are made based on the best information available at the time and they are made in good faith.

Your searing self reflection, your regret, and your integrity are part and parcel of the richness of a life well lived.

Yes, because they were small sections of the tank rather than the full sized complete article. This limited the stresses between the layers of foam so that cracks did not occur. Probably was another aspect of my sloppy writing to call those tests “powerful”

If you look at the findings section F4.2-1, F4.2-2 and F4.2-3, they make it very clear why they make this recommendation (and failure is not mentioned as the cause.) The new design was not tested. It was validated based on an extrapolation from the original design. A small sample size test indicated that the safety factor on the bolt catcher did not meet the 1.4 safety factor. Finally, the QA processes did not meet the contract for a Criticality 1 subsystem. Therefore, the system needed to be certified that it met the safety factor using the actual flight hardware under a QA process that met contract standards for this sub-system.

This is all public domain and takes about a minute to find using Google.

This was in response to Don Beattie’s comment: [i]Sorry to continue my concern, but the CAIB Technical Panel recommendation 4.2.1 concerned SRB bolt catchers. “Test and qualify (new) flight hardware bolt catchers.” What was the reason for this recommendation?[/i]

I believe the original discussion was about the hold down post bolts and an inaccurate depiction of hardware left on the launch pad. The CAIB recommendation and your comment are about the bolt catchers used during SRB separation, an event which takes place later in the launch phase than when the foam was liberated on Columbia or in this case for Discovery on STS-114

Sorry for the confusion. I wasn’t responding to the post about hardware left on the launch pad, though I think it led to the question I was responding to. This was supposed to be a reply to Don Beattie’s question above about “why the panel made the recommendation to test and qualify (new) flight hardware bolt catchers”, indicating that he suspected it played a role? I was pointing out that the CAIB report documented why they made that recommendation in the Findings section. As you say, it had nothing to do with the foam that was liberated on Columbia or Discovery STS-114. It had to do with the fact that during the investigation they found a critical sub-system had not gone through proper QA after a redesign.

Thanks for this piece of history. It proves that without humility, learning is impossible. One question, probably naive: Any reason why NASA never fitted a launch shield on the shuttle’s underside to protect heat shield and flight surfaces from debris damage?

Thanks for the explanation. My engineering knowledge is rudimentary: If it moves too much, duct tape it; If it doesn’t move enough, spray it with WD-40. My larger concern is the lack of real plan to get man back in space. As I write this, I am looking at a set of Shuttle Pins and a photo of the STS 107 Crew, so you know where I am coming from. If there is no continuity, the lessons learned and knowledge acquired (all at a very dear price) have nowhere to go. I am sure the succeeding iteration of Shuttle 2.0 would have been damned impressive. Instead we are reduced to hitching rides with the Russians, while the equivalent dollars that could have unleashed another wave of innovation, discovery and achievement slide through our hands, building nothing, teaching nothing, delivering nothing to the American people. Damned right I am mad about it.

For those who obviously don’t know Mr. Hale, who may have criticized his decisions during the Space Shuttle Program (SSP) days (especially after reading this blog), and who have attempted to minimize his credentials because of erroneous spelling and/or grammar, I feel compelled to set the record straight: As the SSP Manager, Wayne led the Program with the utmost integrity and precision. (Note that I don’t claim he, or anyone else, was perfect.) Of the leaders I have known, Wayne is the most disciplined yet humble. He was a manager who, without reservation, accepted accountability when mistakes were made, whether directly or only organizationally responsible. (And if you don’t believe my characterization of his accountability, then explain the reason he wrote this blog.) During his NASA career, Wayne truly inspired a generation of people and created immeasurable value to the Mission. I will grant to those critical about his grammar that this specific blog may not be his best in those terms. But, the message is enormous–which is typical of Wayne. Regardless, as a NASA manager, Mr. Hale was an excellent communicator. Listen to or read from the people he touched over the years about his lecturing, rants, team building pep talks, or carefully crafted notes. He is easily one of the most intelligent persons I know. (No matter the number or description of degrees Mr. Hale may have hanging on his wall, he is intelligent enough to grasp just about any concept or theory that can be thrown at him. He can think well outside the “Mechanical Engineering” box.) His accomplishments and impacts to Manned Space Flight, and the NASA family of people, are enormous and historic.

I would also add that NASA never sold tickets for passengers to fly on the Space Shuttles. These were experimental vehicles, as meticulous and rigorous as they were designed, tested, and operated. And as routine as most flights appeared to the unfamiliar, every flight was another experiment, too. Moreover, these were experiments of more than just machine and physics. These were trials of immensely complicated working relationships with people, teams, organizations, private companies, governments, political groups, experts, and personalities. Perfection and complete understanding was never possible, and will never be for any future vehicle/program. No problem was ever simple, and no solution ever easy or without unintended consequences. The event that occurred as Wayne described it here is not unlike events that occur in other large, complex programs except that the outcome was considerably more visible and could have critically affected the lives of people the NASA family worked with and deeply cared for. As participants here, we should attempt to discover all that can be realized from Wayne’s account and learn to be more reasonable in criticizing our colleagues.

I’m a fellow shuttle-hugger, but reading this dispelled any doubt in my mind that it was time to retire those magnificent vehicles from service. The tragedy is that we didn’t have the national will and vision to develop an improved second-generation shuttle in the 80s.

For everyone who keeps bringing back up the idea of reinforcing the foam, the problem is that the foam is very lightweight and also fragile. But so lightweight that most of the time the foam holds together. Anything that increases it’s strength, also increases it’s weight to a level that under the stresses of launch the foam breaks off even worse, and with increased mass, would cause more damage. If you get to a level of reinforcement that can withstand the accelleration, vibration, and supersonic wind, you no longer have much insulating effect from the foam, and have added a lot of weight.

Wayne: I’ve wondered if there were any changes that could have been made to the Columbia’s reentry flight. Longer deorbit burn to reduce speed before atmospheric interface, turning the undamaged wing into the wind, etc?

Investigation and analysis continued after the CAIB report was published and we established that none of those makeshift repairs would have been effective. The only long shot possibility was a rescue with Atlantis and that would have been very difficult to pull off.

So Wayne, I understand your story, and that we almost lost Discovery. But does it mean that we almost lost the crew as well as you indicate? Because I was under the impression that 1) the inspection capability would defnitely have determined beyond a doubt had there been catastrophic damage and 2) had it found catastrophic damage, the crew would have remained at ISS and would have been deorbited via Soyuz. So we would have lost Discovery but not the crew. If that is true, then credit is deserved for building in that redundnat means of saving the crew as protection against our own shortcomings in all that was done following Columbia. We knew enough to know that we should never trust that we know enough, and I think that was a lesson learned — and relearned — during shuttle.

Great read. Obviously it was probably not possible to do a tank redesign from the ground up due to costs and time, nor was the technology available in the 70s when the Shuttle was first designed. But I wonder if using composites today it would be possible to design a tank that was light enough to incorporate a dual wall design with the insulation encapsulated therein? That seems like the only way to truly solve the problem. I agree, foam is foam. It’s just not possible to completely control such a material on the outside of the tank.

Wayne: Sorry to waste your time beating a dead horse. The fact that the shuttle is no longer flying drives me around the bend. There were no good reasons to retire it. Just a legacy of Mike Griffin’s ego and poor congressional oversight. As perhaps you know a NASA panel had judged it could continue to fly safely until 2020 after FRRs that complied with the CAIB’s recommendations. You, your team and those that followed only have my admiration for safely launching a very complicated, amazing machine. It should still be flying instead of anchored to the ground and being gawked at by curious museum visitors. Cheers

Wayne can answer this better but the STS retirement was partially because the CAIB requirements were difficult or costly (if memory serves). To recertify each orbiter, it would be very costly. Pardon me for not giving the details as it’s been a while since reading about those decisions. Maybe Wayne will need to correct this, but my understanding was there were concerns regarding the Kapton wiring and the composite tanks. An engineer friend and I argue this often because he sees the orbiters as ‘young’ (very few operation cycles) and I believe the age of some systems are the key not the cycles. The same systems that CAIB wanted recertified.

We were essentially complete with STS recertification by mid 2009. Kapton wiring was a problem but mitigations were in place to prevent problems. The STS was retired not because of the CAIB requirement for recertification but because of the fundamental risk and cost of the shuttle program.

What do you think of the SpaceX Dragon capsule? I won’t rehearse its capabilities, but it seems to have much to recommend it, including robust self-propulsion and potentially vertical dry-land landing capability.

I agree. Wayne, I don’t know if you knew Craig Steidle or not at NASA, but he was one of the associated administrators who did a lot with the early stages of the Orion CEV project before “retiring” to go teach at the Naval Academy (where I had him for several of my Aero Enginering classes). He hated the Ares program though and thought that certifying the Delta IV would have been a much more cost/time effective way to return us to space after the shuttle’s retirement. Do you feel that that option would still be a quicker path to space than waiting on the commercial sector to develop a reliable, man-rated rocket?

Cryogenic fueled rockets frequently form ice on the outside. Look at old movies of the Saturn V launches and watch all the ice falling from the rocket as it lifts off. The early shuttle designers knew that ice would be hazardous. So they put enough foam insulation on the outside of the tank to keep ice from forming under almost all conditions. That is why the foam was there.

Nothing about UFO’s. Just discussing the “ever-worrying” nature of real Engineers, their (Our) tendency to always think that we could have done better, and the disparity between risks perceived by Engineers and technicians, vs the risks portrayed by “management”.

Sorry then, I am just completely overrun by the number of comments on this posting. So many of them seem to repeating others that I have started only approving those comments which don’t already seem to have been made.

Wayne,
I concur with the earlier post that you were the best NASA ever had at briefing the public – particularly in turning NASA jargon into everyday real language that almost everyone could understand. I always will be a fan, but I’m also a fifty-something person who can certainly relate to the inability to poke at those tiny cell keyboards. Please keep sharing and all the best!

Thanks for that response. It is ironic, but it seems in the end America as a nation chose to deliberately lose Discovery and have its crews take the “defense in depth” approach for the indeterminate future.

Hello Wayne.
Thank you so much for your hard work in engineering and as shuttle program manager. I have followed your news conferences in detail via NASA TV, and other internet sources since Return to Flight STS-114, and I’ve always found your words to be quite educational, eye-opening, and inspirational. One example is (if I remember correctly a day at MIT) and you spoke in detail, about human errors and redundancy, and made the statement, “the only real errors are human errors.” It is a statement that I keep with me everyday and will never forget.
Finally, thanks for the update on Discovery. I will be seeing Discovery soon at her new home, and I will not forget all the hard work and dedication put into the shuttle. It’s been an amazing era of discoveries and Discovery helped make it possible!

Your blog says: “… a side note in a final briefing before Discovery’s flight pointed out that the large chunk of foam that brought down Columbia could not have been liberated from an internal installation defect.” All the same, you decided to fly. Arguably that decision flew in the face of the CAIB report, and provided another example of NASA managers willing to overlook problems in order to maintain schedule. As you said, “We were exhausted with study, test, redesign. We decided to fly.” Seems to me you had both a reason and a mandate to postpone return-to-flight.

Wayne, I’m an old Saturn V 1st stage fabrication tech, and S-II second stage propulsion systems inspector. I worked at Michoud, and MTF, now called the John C. Stennis Space Center. In reading your comments about the shuttle as well as the various comments by your readers, it is a relatively unknown value to most people that “what I am working on today will carry someone into space and bring them back safely,” requires extremely focused attention to all kinds of insignificant details that can wreak all kinds of havoc.

In the S-II program, circa 1965, I was sent from MTF to Seal Beach, CA to observe the work being completed on the S-II-T bird, a fully functional bird, that was built to test all the fabrication tooling, transportation, and test facilities, and then become a fully assembled Saturn V as KSC to test out to launch facilities. With less than 2 weeks of employment with North American Aviation(who built that stage) I was sent to CA with no technical briefing or information about the stage. My first night on the job was to “check on the work going on up in the top of the hydrogen tank and forward skirt area, which was considered a “clean room environment” with everyone wearing the “bunny suits & caps.” To my dismay I kept hearing “Tinkle, Tinkle, sssssssshhhhhhhhhhh.” In a couple of hours I finally saw the culprit, a stainless steel locking clip ring sliding down the slick surface of the tank bulkhead. These were used to keep the ground support equipment bolts from falling off when the pieces were removed. This was a piece the size of a paper clip, straightened out, then cut in half, then form the 1/2 into a 4 pointed star, no bigger than that. I had heard about a dozen of these events so I wrote up a discrepancy report saying they needed to be removed from the area “over the horizon” where it couldn/t be seen where they slid down to the bulkhead/sidewall junction area. That shut down all the work going on in that area, and my co-worker found a non LOX-clean environment in the J-2 engine area, so with 2 pieces of paper we shut the whole job down, and we were hated for it! 3 months later, after it had been installed in the A-2 test stand at MTF, the engineers came and asked me what we needed to do about it; I then showed them in the drawings the very tight clearance between the bulkhead and the sidewall, showing them that those “little bitty things” would fall down even further when it was filled LH2, and after tanking, or during firing they would cause unknown stresses on the fairly thin metal involved in that area. They finally came back a month later, wrapped 100′ of X-Ray film around the entire stage and shot a picture. What they found out was over 200 of those “little bitty things,” plus pennies, nickels, dimes, quarters, plastic combs and ballpoint pens, plus a 1/2 X 9/16ths wrench, and a 10 lb piece of steel called a ‘bucking bar” for upsetting rivets! This was literally sabotage, which brought the FBI in to put a plastic sheet all the way around the stage, remove all the super-high-tech insulation as well as the 900 bolts, then as the forward skirt was lifted up even dollar bills came out that didn’t show up in the X-Rays! That put a heavy dent in the testing schedule, meaning money wasn’t paid for a milestone not reached. Yet at the same time as they were raising the skirt and we saw all this stuff falling out, my quality control supervisor turned and looked at me with pure hatred in his eyes! By speaking up for the “little bitty thing” I literally saved the Saturn program. Yes, they went back and did a whole review of that and more with correcting those faults.

Do you remember the Apollo I disaster? That resulted in NASA coming out with their PRIDE program, which I had some initial doubts in what that would accomplish, but when the acronym was explained as:
P= Personal
R=Responsibility
I=In
D=Daily
E=Effort

Look what happened during Apollo 13!

Wayne, these are things that go on, and on, and on, literally everywhere, not just space flight or aviation, but in so many of our everday activities, as I found out in my 30 years in the oilfield after leaving that program. Too many people are doing their work “just good enough.” not with the pursuit of excellence, which is what is really needed if you are to get any real satisfaction in whatever line of work a person may be involved. I would hope that NASA would be more proactive about making some of these little problems known to the public, so we can appreciate what is really required to achieve these remarkable advances. We can never take these newer programs that will be coming into reality, (real soon I hope!) as each one of them will challenge us in new and unknow ways and areas we have yet to learn about. So thanks for your discussions about the little things that can get out of hand and HAVE to be overcome. I hope your readers will have a more forgiving and understanding attitude toward these people that have to manage(and actually build, test, and fly!) these programs.

Wayne, I’m just wondering if the insulation that caused extensive tile damage to the classified STS-27 mission back in 1988 the same (or similar) “light and airy” foam material that ultimately doomed Columbia?

The nose cap was not ablative, it was reinforced carbon carbon material and was reused for all flights; I don’t ever recall replacing a nose cap although we did a lot of inspections. Temps could reach over 3,000 deg F.

We removed foam on foam applications to the maximum extent possible; it was the difference in the coefficient of thermal expansion between the different layers (of different formulations) that caused the cracking. And we never eliminated the threat, only mitigated it.

A quick “thank you” for this glimpse behind the scenes. I followed STS-114 very closely through NASA TV and vividly remember this, and also remember your candor during the press briefings about the foam loss and related risks. The depth of your concern was palpable even through the television, so I’m grateful to have this part of the story.

Thank you not only for this recollection, but for sharing this blog with us all. It’s taught me a lot.

Hi Wayne, it is perhaps moot by now, but what do you think of the idea of a lightweight fiber net (fish net -like), that stays in the foam to hold it together? like in rebar concrete? I Imagine ways can be found to uniformly stand the net off the tank before the foam is spayed on, so to avoid the net-in-between-foam layer problem?

I don’t want to be terse, but I seem to be answering that question over and over and over again.
This idea was suggested multiple times, investigated, and we could not make it work.
Please read some of the earlier comments and their replies for more details

Wayne, is there any link between this testing and the early cancellation of Constellation? Is there any possibility or scenario, since the technology is lighter, more effective and efficient, that we might re-light an STS for emergency payload delivery or extraction, I mean to say, ANY possibility? What will NASA do with all the hardware and equipment manufactured for an STS that now sit in their climate controlled warehouses? Finally, you have done well sir, very well.

I don’t think this links to the cancellation of Constellation, but who knows.
The Shuttle systems have been stripped of flight components to make them safe for display. Nothing is impossible but it would be extremely expensive and time consuming to ever fly them again.

One more thing, as I read the comments, I see that some are not taking your blog as just a piece of information but rather the complete story. I am certain that the story would fill volumes that no one reading this blog would have time to read, or write for that matter.

Well Mr Hale, any chance you would write a book on return to flight? I know I would buy it.

Just read this today, glad it was now and not before I was at the nose to nose with COlonel Collins there but al in all good that I did. Interesting though that the Orion sidewall and X-37B still use orbiter derived TPS (the TuFI tiles, used on the MPS to shield from SSME heat I believe) and EFT-1 will use a Delta IV heavy that uses the spray on foam. That goes to show it is never one component solely to blame, it is the system.

Thank you for your service, and hopefully one day you can come over to Virginia and see lucky Discovery again.

Wayne: let me just say again what a fantastic blog you’ve got, what a great story you’ve told, and what excellent (and patient) comments you’ve given to us readers. Regarding foam on the ET, I now understand that its purpose was maybe not so much to keep the contents COLD, as to keep the water in the air outside the tank from condensing and forming ice. Makes me wonder if those goals could have been achieved with a coating that was deliberately shed just before liftoff. Like a big wine-cooler sleeve that would be peeled off at T minus 30 or something… And if you’re fed up with listening to amateurs yap about foam, I’ll understand. Again, thanks.

Wayne: This was the reason for my question: Cheers
4.2 Remaining Factors
Several significant factors caught the attention of the Board during the investigation. Although it appears that they were not causal in the STS-107 accident, they are presented here for completeness.

Solid Rocket Booster Bolt Catchers

The fault tree review brought to light a significant problem with the Solid Rocket Booster bolt catchers. Each Solid Rocket Booster is connected to the External Tank by four separation bolts: three at the bottom plus a larger one at the top that weighs approximately 65 pounds.These larger upper (or “forward”) separation bolts (one on each Solid Rocket Booster) and their associated bolt catchers on the External Tank provoked a great deal of Board scrutiny.

About two minutes after launch, the firing of pyrotechnic charges breaks each forward separation bolt into two pieces, allowing the spent Solid Rocket Boosters to separate from the External Tank (see Figure 4.2-1). Two “bolt catchers” on the External Tank each trap the upper half of a fired separation bolt, while the lower half stays attached to the Solid Rocket Booster.As a result, both halves are kept from flying free of the assembly and potentially hitting the Orbiter. Bolt catchers have a domed aluminum cover containing an aluminum honeycomb matrix that absorbs the fired bolts energy. The two upper bolt halves and their respective catchers subsequently remain connected to the External Tank, which burns up on re-entry, while the lower halves stay with the Solid Rocket Boosters that are recovered from the ocean.

If one of the bolt catchers failed during STS-107, the resulting debris could have damaged Columbias wing leading edge. Concerns that the bolt catchers may have failed, causing metal debris to ricochet toward the Orbiter, arose because the configuration of the bolt catchers used on Shuttle missions differs in important ways from the design used in

initial qualification tests.1 First, the attachments that currently hold bolt catchers in place use bolts threaded into inserts rather than through-bolts. Second, the test design included neither the Super Lightweight Ablative material applied to the bolt catcher apparatus for thermal protection, nor the aluminum honeycomb configuration currently used. Also, during these initial tests, temperature and pressure readings for the bolt firings were not recorded.

Instead of conducting additional tests to correct for these discrepancies, NASA engineers qualified the flight design configuration using a process called “analysis and similarity.” The flight configuration was validated using extrapolated test data and redesign specifications rather than direct testing. This means that NASAs rationale for considering bolt catchers to be safe for flight is based on limited data from testing 24 years ago on a model that differs significantly from the current design.

Due to these testing deficiencies, the Board recognized that bolt catchers could have played a role in damaging Columbias left wing. The aluminum dome could have failed catastrophically, ablative coating could have come off in large quantities, or the device could have failed to hold to its mount point on the External Tank. To determine whether bolt catchers should be eliminated as a source of debris, investigators conducted tests to establish a performance baseline for bolt catchers in their current configuration and also reviewed radar data to see whether bolt catcher failure could be observed. The results had serious implications: Every bolt catcher tested failed well below the expected load range of 68,000 pounds. In one test, a bolt catcher failed at 44,000 pounds, which was two percent below the 46,000 pounds generated by a fired separation bolt. This means that the force at which a separation bolt is predicted to come apart during flight could exceed the bolt catchers ability to safely capture the bolt. If these results are consistent with further tests, the factor of safety for the bolt catcher system would be 0.956 ｭ far below the design requirement of 1.4 (that is, able to withstand 1.4 times the maximum load ever expected in operation).

Every bolt catcher must be inspected (via X-ray) as a final step in the manufacturing process to ensure specification compliance. There are specific requirements for film type/ quality to allow sufficient visibility of weld quality (where the dome is mated to the mounting flange) and reveal any flaws.There is also a requirement to archive the film for several years after the hardware has been used. The manufacturer is required to evaluate the film, and a Defense Contract Management Agency representative certifies that requirements have been met. The substandard performance of the Summa bolt catchers tested by NASA at Marshall Space Flight Center and subsequent investigation revealed that the contractors use of film failed to meet quality requirements and, because of this questionable quality, there were “probable” weld defects in most of the archived film. Film of STS-107s bolt catchers (serial numbers 1 and 19, both Summa-manufactured), was also determined to be substandard with “probable” weld defects (cracks, porosity, lack of penetration) on number 1 (left Solid Rocket Booster to External Tank attach point). Number 19 appeared adequate, though the substandard film quality leaves some doubt.

Further investigation revealed that a lack of qualified non-destructive inspection technicians and differing interpretations of inspection requirements contributed to this oversight. United Space Alliance, NASAs agent in procuring bolt catchers, exercises limited process oversight and delegates actual contract compliance verification to the Defense Contract Management Agency. The Defense Contract Management Agency interpreted its responsibility as limited to certifying compliance with the requirement for X-ray inspections. Since neither the Defense Contract Management Agency nor United Space Alliance had a resident non-destructive inspection specialist, they could not read the X-ray film or certify the weld. Consequently, the required inspections of weld quality and end-item certification were not properly performed. Inadequate oversight and confusion over the requirement on the parts of NASA, United Space Alliance, and the Defense Contract ManagementAgency all contributed to this problem.

In addition, STS-107 radar data from the U.S. Air Force Eastern Range tracking system identified an object with a radar cross-section consistent with a bolt catcher departing the Shuttle stack at the time of Solid Rocket Booster separation. The resolution of the radar return was not sufficient to definitively identify the object. However, an object that has about the same radar signature as a bolt catcher was seen on at least five other Shuttle missions. Debris shedding during Solid Rocket Booster separation is not an unusual event. However, the size of this object indicated that it could be a potential threat if it came close to the Orbiter after coming off the stack.

Although bolt catchers can be neither definitively excluded nor included as a potential cause of left wing damage to Columbia, the impact of such a large object would likely have registered on the Shuttle stacks sensors. The indefinite data at the time of Solid Rocket Booster separation, in tandem with overwhelming evidence related to the foam debris strike, leads the Board to conclude that bolt catchers are unlikely to have been involved in the accident.

I was going to chew on you for having the chutzpah to lay ALL of the blame for the loss of Columbia at the feet of the minions in Michoud, but then you say that you went back down there and ate your words.
Were there any suicides between the two events? You laid one hell of an unfair burden on those people!

In the years since, your late January/early February posts indicate that you’d learned that you live as a team and you die as a team.

At least the NASA Family stuck with it and, like the ET stringers cracking, determined what the actual root cause of the foam losses was.
The price of spaceflight safety, like that of freedom itself, is eternal vigilance.

I was there on Thursday. If we’d lost Discovery on the pad because of defective stringers, I wouldn’t have been able to live with myself. So, it was worth 100 miles of rain on a motorcycle on Wednesday to be there when Discovery and Enterprise had their “Golden Spike” moment.

Thanks for your honesty in sharing this. From your blog: “What you probably don’t know is that a side note in a final briefing before Discovery’s flight pointed out that the large chunk of foam that brought down Columbia could not have been liberated from an internal installation defect. Hmm. After 26 months of work, nobody knew how to address that little statement. Of course we had fixed everything. What else could there be? What else could we do? ”

Sorry Wayne, it seems to me that you launched knowing there was an unresolved problem, not unlike the Challenger accident decision. What else could you do? Ground the vehicle until the problem is fixed!! The crews’ lives and the future of NASA was at stake.

I always wondered why the polyurethane based spray foam was used over and over expecting a different result, given its coefficient of expansion (CTE) is very large and it goes through a transition at cryo-temperatures where its elastic modulus becomes very large – both characteristics leading to very high foam tensile stress interacting with the aluminum substrate. A fix would have been to substitute polyetherimide (PEI) foam only in those locations where cracked foam might find its way to critical orbiter components. PEI foam is stronger than polyurethane foam, its CTE is a fraction of that of polyurethane, and its elastic modulus (at cryo temperatures) is much lower than polyurethane (it does not go through a transition), all leading to a robust local solution. The only drawback is PEI foam cannot be sprayed, it must be machined or heat formed to shape, and bonded on with FM-300-2 film adhesive at 250F.

By the way, as a contractor, I had the opportunity to work on technology development with TPS engineers from JSC and LaRC, and cryotank engineers from MSFC and LaRC in the late 90’s and early 00’s. Their technical excellence, professionalism, and advice always made my job a lot easier. You were in good company.

“Two major types of foam have been used on the ET. The first type is an ‘acreage’ material that is automatically sprayed on in a controlled environment. This material is designated as NCFI 24-124. It replaced CPR 488 in 1998 and has been used since that time. The other major foam is a ‘hand-sprayed’ foam that is used to ‘close out‘ regions where the various sections of the ET are attached. Two types of close-out foams have been used on the ET, known as BX-250 or BX-265, depending on when the ET was built. The external tank designated as ET-116 was the first tank to use BX-265 in place of BX-250, however, ET-93 did have some rework done with BX 265 on the fairing over the aft ET/Solid Rocket Booster (SRB) fitting. The bipod-ramp foam that was lost during the ascent of Columbia was made of the BX 250 material.”

“The NCFI 24-124 foam is primarily a polyisocyanurate.”

“The BX 250 and BX 265 foams are primarily polyurethane”

“This report concentrates on BX 250 and BX 265 polyurethane foams that are used for the manual close-out sprays because these materials have been identified as the source of most of the observed ‘foam shedding.’ “

This is an amazing story, I can’t imagine under what pressure you must have been under in those days. It goes to show your honor that you made it through as moral person.

Your story reminded me of an experience I had as a kid, petty in comparison. At the age of 13 or so, I used to build small rockets out of paper and wood, proppeled by small gunpowder engines.
For one of the last rockets I built, I decided to equip it with some fancy looking paintjob, something I never bothered to do with my previous designs. I took some synthetic paints I found at home, black and white, and painted that checkered pattern you could see on the old pictures of experimental V2s. I spent a lot of time on the job, and because I was using templates, I decided to even out the surface with another covering layer of transparent paint that I could sand down later. I went with thing called nitrocellulose paint. I was sensinle enough to test how the two paints go together on a small piece of wood. They seemed to be doing quite well together.
I didnt know that at a time, but apparently putting organic paint on top of synthetic one is really bad idea. All the paint on the rocket crumbled as if to fall of, whole thing ready to be thrown away. The area I used to test the paints together was too small for the tensions that arise between the two paints to become apparent.
Nothing groundbreaking here, but I think it shows well as well how the scale of the test objects to the real think can matter in some cases.

MDAC S-lVB had the insulation on the inside with no problems when MDAC proposal for the tank was reviewed by NASA we were rejected for cost, it was better to use a starfoam cup design, but this design was flawed so the results were bad on picking low cost over safety.

Its’ interesting to note that while the tank was certified on paper to something like 11 or 13 thermal (fill/drain) cycles, no tank (to my knowledge) had physically been filled and drained that many times with even a scatter factor of 1.0x (where 2x or 4x is commonly used) nor had its foam been inspected at the end of such a cycle life. To me, the lesson here is that the hardware wasn’t properly certified – including appropriate margin. An expensive and instrinsically hazardous operation/test in itself, no doubt, but one that may have avoided the loss of Columbia.

I have been intimately & thoroughly involved in Aerospace (& non-A) evaluation, analysis, and testing for about 50 years. NASA does the best overall evaluation of the available data and arrives at the best solution when they are given thorough results of properly preformed tests which are preformed using suficienty tested components, tested as part of the larger structure to which it is mounted in service. The fallacy is that a majority of the testing is performed on the component without the benefit of the component’s mechanical and electrical and pneumatic interfaces.
This should totally void the test, but seldom does.
The same is true for analytical evaluations. Analysis is only as good as its inputs and assumptions, which are seldom complete or accurate. And this inaccuracy is not necessarily a bad thing. The primary purpose of analysis is to determine loads under various assumptions. So If the assumptions are explicitly defined and published, the analyst can vary the range of each variable. Only when this is properly done, then the analyst can arrive at a Reliable conclusion.
The same is true for mechanical testing (vibration, structural, fatigue, thermal,,,etc). Here, the weakest link to reality is the mechanical connection between the speciman and the lab-simulated mechanical interface, necessary to input the correct environment but seldom matches the correct mechanical impedence.

Therefore: A thorough analysis based on specific environmentally tailored inputs and which publishes all assumptions, is a valuable tool for range studies.
An adequate test, based on proper, tailored inputs, and the correct mechanical or electrical or pneumatic interfaces, yields valuable perception as to the critical flight responses.
But a thorough analysis which correlates with a reliable test will yield the best, useful, reliable results.

Mr. Hale, your decision process, your communications to all of us at NASA and at the contractors, and your integrity, is deserving of what some call the good life. The problem is that people with integrity seldom believe that all the decisions have been satisfactorily closed and therefore their life and their responsibilites are on-going; completre retirement is for those without responsibility.
NASA’s history shows that Mr. Hale made good decisions based on the best information available, and made in good faith. Thus justifiying the definition of a rich, well-lived life. And those of us that know him, concur without hessitation.

I have always had a question about the Columbia issue – the foam does not weigh much, correct?
The foam is moving at the same speed as the entire stack obviously.
The air is pretty thin where these foam events have occured so there cannot be all that much deceleration . . .
The foam comes off – the Shuttle is still accelerating obviously – but how can there be that much delta in the foam strike to cause such damage?
I ran the numbers once and given the really short period of time – less than a second – there is less than a 3g acceleration at the point the foam came off – so there is less than 96fps delta v. Thats only about a 70mph relative strike force . .

Can you explain that better for me – I’ve been wondering since 2003 about how such a fairly benign impact can cause such extensive damage – is the tile section that fragile? I understand there is no planning for impact for the objects – other than air molecules – but the impact itself could not have been that intense –

You have a bad assumption that there was no air at the altitude where the foam came off. Even at that altitude there was sufficient air to cause drag on the lightweight foam debris and decelerate it almost to zero velocity very quickly. Then the orbiter, moving at a high rate of speed, ran into the foam debris.

Also, as I understand it, the mass of the foam is only part of the issue. The foam has an air column above it. The physics are out of my league, but the energy behind the impact of a pound of foam with an air column sitting on it must be different than the same impact in a vacuum.

That’s kind of the point I was making – I did not say there was no air pressure – I said the air was thin [ner than sea level] – it is not a vacuum at 66,000 feet – but it is only about 0.82PSI = that is NOTHING in terms of generating air resistance . . .

Did you actually perform the computations as to the strike energy of a piece of foam striking the Orbiter? Or is it one of those ‘gee, of course its gonna be hard?’ assumptions.

I simply cannot, in the span of less than a second, see the foam decelerate to [as you claim] stationary –
I ask this because every single other time we have actually seen foam strike the Orbiters there was no damage. One would think that if the events caused damage from the physics of them that they would cause damage all of the time.

I could do a redneck computation myself if I knew the mass and average surface area of the piece of foam – but it would have to be substantial to come off to cause the level of damage – which then is counter intuitive since a larger piece has more mass, and the thin air would decelerate it less –

These are significant questions because what if the foam was NOT the cause?? Would we not have seen foam embedded in parts of the frame of the shuttle when it burned up? The foam would be traveling the same speed as the Orbiter until it detaches itself, and from the other foam pieces that have come off and the imagery we have – it does NOT look like that significant an impact, e.g., the foam does not disintegrate or come apart – so the Delta V is really not that great . . . I’ve always thought there was basic physics problem with the Columbia foam theory . . .

You write: “The Shuttle systems have been stripped of flight components to make them safe for display.”

What flight systems needed to be stripped? The propellants, obviously, and batteries, but it looks like you meant much more than that. I’m just curious; not in the least suggesting it would be easy (or desirable) to return the vehicles to flight.

All the contaminated propellant systems, the pyrotechnic devices, the comsec gear, the fuel cells, and many parts which might be used on future spacecraft including the entire Main Propulsion System plumbing from Atlantis.

Having watched every episode of Air Crash Investigation, it’s plain that many conscientious competent engineers can simply get things wrong. The mark of a professional is to admit those mistakes and learn from them when there are probably many examples of less professional people who elect to cover up the mistake thus increasing the risk of the same mistake recurring.

“So don’t question whether we were motivated to prevent another loss.”

I don’t think anyone questions whether NASA was motivated to keep their launch systems safe. But I think we learned from the Rogers Commission that the incentives are not aligned quite right: people who have final say (and know the astronauts personally) are not always those who have the most technical knowledge about the situation.

I’m not saying this to call you dumb, because I sure don’t know the answer, either. I don’t know if anyone does. Putting 25 tons of metal (and half a dozen people) safely into orbit is a huge undertaking, which inherently requires the efforts of thousands of people, and that means bureaucracy, and that means lots of possible communication (and mis-communication) channels, and lots of different opinions on the risks faced.

This is the same problem facing many fields today, including my own. The software industry, certainly, hasn’t figured out how to organize thousands of programmers well.

You acted accordingly and made sure lessons were not only learned by immediately applied, you reabsorbed the blame correctly, and we now have a technical and heart-felt explanation for what transpired during the shuttles greatest hurdle. Thank you for posting this, it helps me understand, as well. I hope you are no longer feeling any pains, you are not to blame, either, we lost loves but sacrifices must sometimes be made of good men, accidentally or otherwise, when truly brave things are attempted.

I think you forgot to delete a paragraph while editing. Here’s an excerpt from the beginning of your post:

Discovery was the shuttle return to flight vehicle after the Challenger was lost; two and a half years were spent from January 28, 1986 until Discovery flew in September 1988. Many improvements were made which resulted in a safe space flight.

Discover was the shuttle return to flight vehicle after Columbia was lost; two and half years were spent from February 1, 2003 until Discovery flew in July 2005. Many improvements were made but safety was not assured.

Wayne, you boys at NASA are doing some of the most incredible things ever done by our species. Mistakes are going to happen. Don;t beat yourself up too much and remember, we all love you guys for the great work you do. Keep the faith brother!

Thanks for the interesting insight into the inner sanctum of Shuttle lore. At the time my suggestion, never submitted was to embed a thin mesh like the giant mono filament drag nets used by Japanese fisherman ( i never tried to calculate the weight) into he surface of the foam as it expanded. I had an experience spraying foam on a 60″ geodesic home I built in 1970. The foam covered the entire exterior 2″ thick creating waterproof and insulating membrane in one fell swoop. Foam loss due to acceleration and temperature differentials was not, fortunately, one of the domes problems. Did anyone suggest the net at the time?

Wayne, I just stumbled across your blog tonight because of a Spaceflight Now post on Facebook… I’ve enjoyed reading back through your posts and can only imagine the stress and pressure of having lives depend on your launches… I’m an engineer for a small aerospace test firm and I’ve gotten to see first hand (on much smaller tanks) the cracking of foam insulation on thin walled cryogenic tanks. I had never connected the dots that that was the issue with the ET foam until reading this. Thanks for writing it!

I wouldn’t question well-intentioned people about their motivations. I don’t question whether 1+ 1 = 2 or not. The questions I have are the following: does NASA management still disregard *both* internal & external criticism? Has anyone *ever* been fired for disregarding safety procedures that were their responsibility? And why should taxpayers believe them?

Foam loss from the External Tank occurred during ground testing prior to the first flight of the shuttle and occurred on every flight thereafter. It was mischaracterized from the start as a nuisance rather than a major risk. Why it was mischaracterized is perhaps a subject for an entire book.

Only today was I introduced to your site and may I say, it’s a nerdasm of space insight. Thank you for taking the time to create and maintain this blog. I have read all the prior comments on this topic yet haven’t seen my question covered. Please bear in mind I am an absolute layman so don’t type too far over my head.

Why was the LOX line external to the tank? Harking back to the days of the Saturn V, the second and third stages used a LOX tunnel through the LH² tank. Why wasn’t this approach taken with the STS ET? If the ET had been a perfect tube would it have been easier to apply the foam in a single layer and less likely to detach?

I’m sort of ignoring the attachment points for the orbiter and SRBs when I suggest the ET would be a perfect tube but I don’t recall reading about any issues with foam detachment from those areas. Besides, only the forward attachment points could cause any debris-strike issues. Provisions would still be needed for the pressurization lines and electronic cabling but again, all that was managed internally in the Saturn V.

Bottom line. Would having a uniformly smooth ET on the orbiter side eliminated the risk of foam loss and damage?