Posted
by
Soulskill
on Monday January 09, 2012 @03:12PM
from the dumb-mistakes-cost-the-most dept.

An anonymous reader writes "The alleged rescue of a U.S. military communications satellite underscores some of the weaknesses in U.S. space efforts. Quoting: 'The seven-ton “AEHF-1,” part of a planned six-satellite constellation meant to support radio communication between far-flung U.S. military units, had been in orbit just one day when the problems began. The satellite started out in a highly-elliptical, temporary orbit. The plan was to use the spacecraft’s on-board engine to boost it to a permanent, geo-stationary orbit. But when the Air Force space operators at Los Angeles Air Force Base activated the engine, nothing happened. The Government Accountability Office would later blame the failure on a rag left inside a fuel line by a Lockheed worker.'"

Seems like the gov't should sue lockheed for failing to deliver the working satellite as contracted.

Hopefully that'll happen (which will probably leave that worker jobless) and we'll get some of our tax dollars back.

Shhh... I can dream!

Lockheed wouldn't piss off their biggest spender. They'll pay back in the form of a "credit" for some kind of services that have the highest margin for Lockheed. The guy who screwed up and his boss will get fired for sure, and then they will have some business analyst examine their QA process and add a little redundancy in the inspection policies. Nothing to see here folks.

Seems like the gov't should sue lockheed for failing to deliver the working satellite as contracted.

Hopefully that'll happen (which will probably leave that worker jobless) and we'll get some of our tax dollars back.

Shhh... I can dream!

Lockheed wouldn't piss off their biggest spender. They'll pay back in the form of a "credit" for some kind of services that have the highest margin for Lockheed. The guy who screwed up and his boss will get fired for sure, and then they will have some business analyst examine their QA process and add a little redundancy in the inspection policies. Nothing to see here folks.

Isn't that what should happen? I mean, when did the world suddenly decide that anytime anyone makes an honest mistake they should be crucified for it forever? If there is restitution for lost funds as well as improvements to try to prevent a repetition of the same problem, shouldn't everyone involved be satisfied? I'm fairly certain that the OP's hope that we all get some kind of tax refund is probably not going to happen, and even if it did, you'd be talking about a few dollars per person at most.

I mean, when did the world suddenly decide that anytime anyone makes an honest mistake they should be crucified for it forever?

Forgetting a rag is an honest mistake. Failing to plan for honest mistakes by implementing the appropriate checks into your process is negligence.

The engineer following the process is not necessarily the person that created the procedure. Also even if a procedure is in place double failures do occur - they are just less likely.

I love the way so many people are willing to judge that a man should or should not be fired based on 3 minutes of reading a slashdot story. Really enhances my faith in human nature. Hope none of you ever sit on a jury. What disciplinary action if any should be faced by various staff involved is something that would require at least weeks of investigation, IF you want to go in that direction and waste the time on a witch hunt instead of just fixing the issue.

Actually that whole 49% figure is misleading. The element of "time" is conveniently left out. 49% of Americans at any given "time" do not pay income tax, not because they are lazy, poor, freeloading citizens but because they are either too young to earn income or retired. In actuality, over 90% of Americans pay income tax at some point in their lifetimes.

More seriously, why wouldn't groundside testing notice that there was a rag in the line?

Yup, why wouldn't it?

Obviously it didn't. Multiple times. In multiple different situations - this isn't the first space mission to be ruined because of something left where it shouldn't have been.

The obvious answer to your question might be because it didn't block anything during testing, so there was the appearance of nothing wrong. Turn on the fuel flow, after the experience of the launch, and it might have been jostled free from wherever it had chosen to hide - from there it might be a short ride to a bottle necking point such as a crimp in the line, a sharp bend, or a valve, and thus begins the blockage.

I've dealt with people at the GAO, they have a lot of expertise in house on many subjects, and aren't afraid to seek advice externally to come to decisions. It's one of those branches of government that it's actually nice to have.

Junk left in piping can cause all kinds of problems (like, duh). That's why there are procedures to clean out the lines prior to startup. If the rag was left in oxygen or oxygen-rich piping, well...kaboom.

One space industry insider, who spoke with The Diplomat on condition of anonymity, says lapses like the forgotten rag indicate a lack of experience in the lower ranks of U.S. space contractors. âoeIt was probably a mix all too common in the USAF programs: 80-year-old PhDs and 20-year-old college gra

How do they know a rag was left in the fuel line? Do they have a sensor in the fuel line that checks for the presence of rags?

I don't know about this case, but AFAIR NASA required forms signed in triplicate saying that any tool taken into the shuttle was later removed from it. Perhaps there's similar tracking in this case and a check showed up a rag that wasn't signed out for being removed.

It seems to be a common problem, I'm sure I remember a couple of rocket launches which were blamed on rags in the fuel lines.

If it's a rag small enough to scrub the inside of a fuel line, it could easily go unnoticed on its way onto the assembly platform. Or, if one was too large, it could have been sectioned and still taken out as "one rag." But in any case, signed in or signed out, how hard is it to test whether or not the line is partially or fully plugged? Put a controlled pressure on one end and measure flow rate on the other.

I'm a bit surprised that it wasn't caught purely as a side effect of other procedures... If I were planning on running fuel through something important that might theoretically be contaminated with lubricants/condensation/whatever, I'd strongly consider blowing $5 worth of compressed nitrogen through it until the outflow is clean...

Catching every little thing that might gum up the fuel lines during assembly, testing, and cleaning seems like it could be a genuinely hard problem. Doing a combination pressure test/gas flush seems like it would be a cheap, simple, brute-force solution to that entire class of potential problems...

Actually, stuff gets left in patients because surgeons don't use checklists. Apparently think they have fantastic memories and no-one who is not a higher-qualified surgeon is gonna tell them what to do. However, a recent study was done where surgeons took a left from pilot's books and actually started using checklists - the rate of post-operative complications plummeted. The use of checklists has not yet reached widespread use in surgery, AFAIK. I hope it does soon.

I don't know about this case, but AFAIR NASA required forms signed in triplicate saying that any tool taken into the shuttle was later removed from it.

An audit of procedures after the Challenger disaster revealed that this can actually cause the problem it's attempting to solve. When you require three people to sign off that something has been inspected, one day an inspector is a bit rushed and needs to finish by 4:30 to make it to his child's school play. He figures since two other people will be inspec

No, not paper. These things are assembled in cleanrooms, in which ordinary paper is not allowed, due to the particles/fibers it sheds. Cleanroom cloths are usually lint-free polyester cloth squares about 8 inches on a side, IME.

Moth found trapped between points at Relay # 70, Panel F, of the Mark II Aiken Relay Calculator while it was being tested at Harvard University, 9 September 1947. The operators affixed the moth to the computer log, with the entry: "First actual case of bug being found". They put out the word that they had "debugged" the machine, thus introducing the term "debugging a computer program".In 1988, the log, with the moth still taped by the entry, was in the Naval Surface Warfare Center Computer Museum at Dahlgren, Virginia.

Really wow. That kinda surprises me. Other than track side pit crew work, which generally would not be invasive enough to the point of removing the head why would someone be working so quickly and careless not to turn the engine over by hand once before applying the starter?

Seriously after any job that requires removing the cams from their bearings you should probably be turning the engine through at least on revolution by hand just to make sure everything operating freely.

Most starters aren't strong enough to bust up a wrench or socket. Take out a plug maybe, possibly bend a valve, but in all likelihood, the motor would turn the engine till contact and stop.

That is assuming you are hitting the engin with the starter before hooking up the fuel and plugs. Which is usually a good idea to get the oil pump primed and heads lubricated firing it up.

That said, I have a number of wrenches that could easily fit in a cylinder with the piston at BDC. A GM 350 for instance, has a 4" bore and 3.48" stroke. On the diagonal that gives you over 5 1/4" clearance at BDC, not including the combustion chamber in the head.

Because it's hydrazine. Really nasty stuff no one wants to be shooting off on the ground. I read the link and I think whoever wrote the Air Force Magazine article either took liberties or talked to the only knucklehead around Space Command. For example, "We found things we hadn't seen before, such as a warm up period." Really? This is why every GPS satellite ever launched has pre-heaters for it's hydrazine thrusters? (They're called "cat bed heaters" if you're looking at Reaction Control System teleme

Most car engines have a bore of around 80mm. Enough to easily allow a small wrench to sit flat on the top of the cylinder. You could probably also spin the engine slowly without noticing it. 95mm is not completely uncommon.

I have in my possession a canvas tool-pouch containing 12 ignition wrenches that can easily fit inside the cylinder, and has done so many times over the past 4 decades. That's 12 wrenches at a time, and the pouch that has fit totally inside of cylinders many times. (handy place to temporarily hold things)

Somewhat in your defense, I will admit that there are few times you need that small of a wrench around an engine when the heads are remo

I have assembled zero satellites. But back in my military contracting days, I did the electronics for several military applications and was present when satellites were built. The boosters on those things are fairly small, and the fuel lines tend not to be big enough to stuff into what we think of as a rag. Maybe a cleaning tool or some other implement. I think whomever wrote that was either lazy or didn't fully understand what they were writing about.

> Why has Slashdot suddenly fallen into the trap of "I've never seen one so it can't possibly exist"?

Have we so soon forgotten that us slashdotters come from a variety of backgrounds? For instance, legal articles are often responded to by actual lawyers in this group. There are actual astronomers, actual physicists, actual biologists, and I'm certain, actual rocket scientists, who read and participate in Slashdot. We're not all gamers living in our parent's basement. Although there are some.

You have to be joking or you've forgotten what we're talking about. This is an apogee kick motor (to use GPS parlance) to take a 2 ton space vehicle to a circular 22,000 mile orbit. GPS, having half the orbit and half the weight, has an AKM which is not small. It's huge. It has to be due to the amount of firing it's intended to do. I tried Googling an image to show my point, but unfortunately, the AKM is on the "non-sexy" side of the SV and, hence, no photos. We're not talking about.5 pound stationke

I believe the consensus is that the "rag" is a piece of cloth or cloth-like material designed small enough to fit in the line to clean it out. Most people think "cloth designed to be held by a hand" not "scrap designed to be held on the end of a rod for cleaning gun barrels and fuel lines." The wording was inexact and conjured an idea in most people that's inaccurate. Better wording "A small swatch of cleaning cloth is thought to have been left inside a fuel line" or something like that would have been m

Who puts an engine together without a test fire? Seems to me that some simple checks would have prevented a very big waste of funds and effort. I guess it won't be a total waste if they can learn from it.

Actually, the Hubble mirror isn't supposed to be flat, its shape is a particular function. It was actually manufactured exactly to spec, but the spec was wrong.

Actually,the hubble was spec'd to be a conic constant of p=-.0023, but was polished only to p=-.0139 (i.e. over hyperbolic). The error was due to a problem with the tester. The null reference element was out of position by just over a millimeter. The interesting thing is two other testers reported that the mirror was wrong, but they were ignored because they were not the 'primary' testing instrument. You are correct that it wasn't supposed to be flat, but it definitely wan't built to spec.

Minor piece of historical correction: The mirror was supposed to parabolic, not flat, so measuring it was not quite a simple matter. It was not NASA but a subcontractor who did the actual grinding, and they used as a reference a defective measurement device when grinding it. Since they did not want to re-grind the mirror, they ignored the data from two functioning devices that said the mirror was flawed. NASA for its part did not perform sufficient quality control on the process to notice that only one d

> It reminds me of those surgery horror stories where the surgeon or staff leaves behind clamps and sponges inside the persons body.

Funny you should mention that. I had emergency surgery last year for severe traumatic internal bleeding (won't bore you with the details -- or maybe I already have) and things happened so quickly that they did not have enough time for an instrument inventory. (Apparently it's someone's job to keep track of how many tools get used and then count them before final suture.) So after they got me stable they ran me back through x-ray to look for stuff. Didn't find anything, fortunately.

But really -- it's not that much of a horror story, they just have to open you back up at some point to retrieve the objects. It's not something you want to have happen, but it's a fairly well known procedure. Horror stories to me are things like taking off the wrong limb [1] or prescribing catastrophically wrong medication.

[1] Before I went in for knee surgery, the doctor gave me a sharpie and had me mark the correct knee. Just in case.

It does make me curious as to how big the rag and fuel line are. Also makes a great juxtaposition with the story immediately below on the home page, The Challenges Of Building A Mars Base [slashdot.org]!

Actually pre-flight final won't catch that kind of thing; it's already buried in the system (and you don't fire thrusters on a flight unit prior to launch). This is likely one of those cases where a scrap of cleaning"rag" was torn off within the path in an area not visible at either end and went unnoticed. To save money, a visual of the system prior to final assembly was determined to be sufficient and the endoscope procedure was eliminated, saving several thousand dollars (combined on all the lines). Sure, in hindsight a compressed air test would have been sufficient, but it's a little late to play what-if now.

The test was devised, mostly likely. There are hundreds - no, probably tens of thousands - of tests on this craft. You can probably trace every single raw material element back to where it was mined, refined, billeted, shipped, stored, manufactured, machined, binned, tagged, selected, gaged, installed, torqued, tested, and approved for flight. Every single time a human touches a part it costs $100 (well, that was a decade ago, it's probably $200 now).

FTFA"On Oct. 24, AEHF-1 reached its originally planned orbit. Testing began soon afterward. The Air Force expects to bring the satellite into service in March. Meanwhile, two more AEHFs are slated to launch in 2012."

They got it into the correct orbit over two months ago using the small thrusters.In other words...More sensationalistic headlines to get clicks and comments from the new Slashdot.Really? Oh and the answer is "no a dirty rag did not take out a 2 billion dollar commsat."

And of course that "underscores some of the weaknesses in U.S. space efforts." Actually, I would say it underscores the strength: they managed to fix the problem using ingenuity and scarce resources. Also, a "scrap of cloth" != "a rag". Calling it a rag implies someone just forgot a whole piece of cloth. A scrap of cloth implies it ripped or was otherwise accidentally and through no negligence (well, not gross negligence anyways, they may still have checked more carefully) deposited.

I've found this story [thespacereview.com] about US Atlas program in 1993:

Then, three years later, it was the Atlas program’s turn again. On March 25, 1993, an Atlas 1 lifted off from SLC-36B at Cape Canaveral AFS, carrying the first of the US Navy’s new UHF Follow-On communication satellites. The launch proved to an inauspicious start to the new program.

A mere 22 seconds after liftoff the vehicle’s sustainer engine began to lose thrust, ultimately reaching only 65% of its nominal thrust level at T+103 sec. The Centaur second stage performed normally, but was inadequate to the task of making up for the low performance of the sustainer. The payload ended up in an orbit far below the desired geosynchronous transfer orbit. The spacecraft used its own onboard propulsion system to climb to a higher orbit, but one that still proved to be too low to meet mission requirements.

Analysis showed that the sustainer thrust decay was due to a simple problem. The Atlas sustainer engine thrust level was controlled by a regulator that was adjusted by turning a screw. A set screw was to be tightened to ensure that the adjustment screw did not move due to in-flight vibration, and that had not been done properly. The result was another fatal Oops!

FTA: "They didn’t know it at the time, but a fuel line had become clogged. The blockage “was most likely caused by a small piece of cloth inadvertently left in the line during the manufacturing process,” according to the Government Accountability Office." (bolding mine).

So no, we don't know that a dirty rag caused a two billion dollar satellite to fail. We think a fuel line became clogged, and some government bean-counter pulled the dirty-rag hypothesis straight out of their derriere so they could sign off on this one and go home.

FTA: "They didn’t know it at the time, but a fuel line had become clogged. The blockage “was most likely caused by a small piece of cloth inadvertently left in the line during the manufacturing process,” according to the Government Accountability Office." (bolding mine).
So no, we don't know that a dirty rag caused a two billion dollar satellite to fail. We think a fuel line became clogged, and some government bean-counter pulled the dirty-rag hypothesis straight out of their derriere so they could sign off on this one and go home.

The GAO was probably basing its conclusion on statements from Lockheed itself. According to this [bloomberg.com] it was Lockheed that concluded the problem was some cleaning material left in the line.

"It should not have happened,” Deputy Under Secretary of the Air Force for Space Programs Richard McKinney said. “It was a quality mistake and we took steps to make sure it does not happen again,” he said. “It was obviously a very serious error.”

“It appears that there was a blockage in one of the fuel lines,” McKinney said. Lockheed thinks “it was caused by some cleaning material that was used in a line that was not properly vacated when they went through production.”

Please describe to me the process by which you would prove that a fuel line currently in geostationary orbit 24,000 miles above the surface of the Earth has a dirty rag (specifically - As opposed to a dead mouse or a styrofoam peanut, for example) blocking it, without taking it apart and finding said rag.

"Finally, it speaks to the size and age of the U.S. space arsenal that the Air Force felt it had no choice but to rescue AEHF-1 instead of replace it with a back-up spacecraft. 'The asset inventory is getting so tight that they spent months limping the heap to its proper orbit,' the insider lamented."

Look guys, before you throw away (replace with a backup) a $2 Billion satellite, I damn well hope you try some pretty heroic measures. Those are my tax dollars in (the wrong) orbit! So I'm very glad you didn't have (to use) a backup satellite.

Anyway, does anyone know if the low power thrusters which were eventually used to put this satellite into the correct orbit used the same fuel tank as the clogged thruster? Otherwise 1) I'm very surprised they had enough fuel to get there and 2) they would probably have very little left to last the lifetime of the mission. So let's hope that all the thrusters used a central (hydrazine?) fuel tank and there's plenty left.

Space is hard and while the U.S. program has certainly had its ups and downs at least it hasn't seen the near total collapse as what happened to the Ruskies. They had quite a bad year last year and that blogger walking around their factory just exposed their problems more. If Mars is going to be a "Red" planet it will because of China not Russia.

They used the hall effect thrusters instead of the hydrazine/nitrogen tetraoxide engine. The hall effect thrusters run on xenon and electricity, so NO they did not use the same fuel source. The hall effect thrusters have a specific impulse of ~8000s instead of the ~300s for hydrazine, so they are insanely fuel efficient, but extremely low thrust. (1/4N vs ~450N for the main engine)..

Ok.. It appears that they used both. I should have read the whole article.. The hydrazine was expended in a 5N thruster over 12 firings (raising the satelite about 1/7 of the way and changing the inclination), then the 0.25N hall effect thrusters were used for the remaning 19400miles (firing 12hours a day for 8 months)..

A Dirty Rag? C'mon - RTFA! "The blockage 'was most likely caused by a small piece of cloth inadvertently left in the line during the manufacturing process,' according to the Government Accountability Office."

That could mean a tiny fragment of fabric. It's not like they put a rag in the gas tank to keep gas from leaking out. sheesh.

The "main" engine doesn't start so they use thousands of firings of the maneuvering thrusters to circularize the orbit. Do the "main" & maneuvering thrusters use the same fuel source or has the mission longevity been compromised? Does anyone know?

It's not like they have little nanites with cameras crawling around there. Fine, the main thrusters aren't working, but how did they manage to specifically blame it on a piece of rag in a fuel line? Aren't there a lot of ways a thruster can fail to fire?

"Finally, it speaks to the size and age of the U.S. space arsenal that the Air Force felt it had no choice but to rescue AEHF-1 instead of replace it with a back-up spacecraft. &ldquo;The asset inventory is getting so tight that they spent months limping the heap to its proper orbit,&rdquo; the insider lamented. "Translation:

We spent tens, perhaps hundreds of thousands, heck maybe even a couple million on labor to save a $2,000,000,000 dollar satellite rather than build another 2 billion dollar sate

Reminds me about all those stories of bottles put inside cars during assembly. Here's a funny one (albeit fictional):

A man goes to a car dealership one day after inheriting a good deal of money (or after a great business deal, whatever -- he has a lot of money somehow). After looking around the lot, he picks out the nicest, newest, fanciest, most expensive car he can. He pays cash up front and drives out of the dealership in the new car.

On his way home, he starts hearing a rattling sound -- something must be wrong. So he turns around and goes right back to the dealer. The dealer is of course very sorry, and offers to either fix the car or let the man take a different one while they order a replacement. The man really wants the car, so he just has the guy fix it. Two hours later, the mechanics give the car back, saying they couldn't find a thing wrong with it. The man is a bit wary, but he drives home. Whatever the rattle is, it has stopped.

A day or so later, the rattle starts again. He takes it to the dealership, and they still can't find anything wrong with it. This continues for a number of weeks -- sometimes the rattle even goes away on its own. Anyway, after nearly two months of it, the dealer is very upset -- he doesn't want to get a bad reputation. So he orders a replacement and exchanges it with the man for the malfunctional car.

Then he orders the mechanics in the shop to do a complete tear-down to figure out the problem. They begin taking the car apart, piece by piece, but they can't find anything -- until they take apart the door. Inside, they find a piece of metal pipe, along with a note. Written on the note, in a scrawling, worker's hand is: "So, you finally found the rattle, you rich son-of-a-bitch."

There is no possible way that this fellow's mistake was the only one made during the entire production process. There were thousands of mistakes similar to this that went into the satellite in question (there are thousands upon thousands of parts and processes required, after all), this was merely the one that escaped all efforts to eliminate it... Unless you can prove that he really was conspiring to sabotage the satellite, you need to blame the process and not the producer. For all we know, the rag ha

The airlines did it and improved by amazing amounts (nobody remembers how bad it was) and the things were much less complex to fly back in those days; the pilots were insulted by it as well. CHECKLISTS WORK.

Something that important should involve multiple checklists; to error is human no matter how good and smart you are. Doctors are the most arrogant pricks I've ever met so they'll put up a huge fight and have a hard time admitting it when the error rate goes down by half. It likely would go down by half; that is how badly it is needed.

Nurses too... a friend of mine fought off his nurse violently (as much as he had strength post op) she had to call people in to hold him down and sedate him and luckily somebody heard his screams and READ the chart and realized she had the wrong person! he would have died and without a proper autopsy the cause wouldn't have been known. Mistakes killed my father too. Checklists must be mandatory by law like the pilots who have no issue with them today.