Breaking news on the ISEE-3 mission: it may not be lost – it's those "O" rings again

A few days ago, my heart sank when I heard this news: Space Probe Might Lack Nitrogen to Push It Home It seemed a deal-killer if the nitrogen tanks were empty. Now, one of the team leaders has given me an inside track into the issue, and the mission may not be lost, thanks to the collective consciousness of the Internet.

In the science fiction universe of Star Trek, set several hundred years in the future, when we are a spacefaring civilization, humanity encounters a species called the Borg. The Borg are a conglomeration of species who are assimilated into a collective mind numbering in the hundreds of billions. All of the borg are connected to each other through a communications link that allows each of them to share each others thoughts, though in a manner that erases individuality.

This week, with the call that our ISEE-3 reboot team put out to the internet for help in debugging our propulsion system problem, I have come to realize that a significant portion of humanity has reached a Borg like state, one where the internet has become a collective mind for communications and knowledge sharing. We still have our individuality, we can still decouple at will from the collective mind, but in a way that few philosophers or technologists have envisioned, we are connected in a way never before thought possible. The implications are staggering, and here is how our little ISEE-3 project is an example of the operation of the collective mind.

ISEE-3 Trajectory Correction Maneuver Attempt

On July 8th and 9th the ISEE-3 reboot team attempted to fire the radial thrusters on the spacecraft. Figure 1 shows the configuration of the thrusters:

The thruster firing on the 8th was a follow up to the successful spin up maneuver that was performed on July 2nd, 2014. The engine firing on both the 8th and the 9th was unsuccessful in producing the thruster needed to change the course of the spacecraft to what is necessary for the lunar flyby on August 10th. After the largely unsuccessful Trajectory Correction Maneuver (TCM), on the 8th, our team decided on a strategy to both use alternate thrusters and to troubleshoot the propulsion system. On the 9th the TCM was also largely unsuccessful (a few pulses out but no substantial change in course). After a bit of team depression that the mission was over, a failure investigation was started. Following are the results of that failure investigation, and our plan for recovery.

First TCM Attempt July 8th

After the successful July 2nd spin up maneuver we had high hopes for the TCM. We had learned quite a bit about the operation of the spacecraft and the spin up maneuver made us and our NASA partners confident in the next steps. Indeed it was the successful spin up maneuver that allowed NASA to give us permission for the TCM. However, as events would unfold, this confidence was to be misplaced.

As those who have read of our exploits know, the ISEE-3 is a spinning spacecraft. It spins quite rapidly, at 19.75 (19.787 right now) RPM or about one rotation every 3.04 seconds. The thrusters for the propulsion system that allows for an in plane change in velocity are located on panels 1 and 9 (redundant) of the spacecraft. This is shown in figure 2:

Figure 3 shows the schematic of the propulsion system from the same AIAA paper from 1979:

The spacecraft has two completely redundant fuel, latch valve, and thruster systems. The thrusters for dV maneuvers are located on the exterior of the spacecraft around the circumference as shown in figure 2. These are the “radial dV” thrusters. For the maneuver on the 8th we used thruster’s F and N, which are connected to the HPS 1 tank system. Everything proceeded normally as we followed the worksheets provided by our documentation. Things went much better from a commanding perspective as we were able to leave the transmitter and power amplifier in transmit mode at Arecibo while we received telemetry pipelined to us over the internet from the radio telescope at Bochum Germany, which is operated by the AMSAT DL team there.

When we fired the thrusters for the TCM maneuver, initially things looked great, but soon we saw a fall off in thruster performance as monitored by an accelerometer designed for this purpose. This is shown in figure 4:

The upper graph is of the accelerometer output during the three firings. The middle graph is the plot of the Fine Sun Sensor (FSS), which gives orientation. The bottom graph is the telemetry value indicating whether or not the Latch Valve opened or not. At this scale it is hard to see the actual acceleration values and they are in counts, not m/sec squared. It is quite clear the fall off in thrust in the first maneuver. We now know that the latch valve was not opened (we did not then). Later, and before we did the second attempt, we see the sun angle decrease. This so far has not been explained adequately.

The third attempt is the most interesting. We started wondering about the latch valve and then sent the command for it to open again. It opened. The difference is that we did not have the +28 volts on the first time as none of this is in the procedures that we currently have available. After we commanded the latch valve to open with + 28 volts on we saw telemetry confirmation opening. Figure five shows what we saw in detail regarding the FSS sensors:

While this is probably hard to read on a screen but figure six is toward the end here when we first got confirmation that the latch valve between the tanks and the propellant lines was opened:

The latch valve opens at 9200 seconds and there is an immediate effect on the accelerometer and the FSS sensor. The data rate is 8 hz for the accelerometer and 1/8th hz for the FSS sensor so there is a lot of aliasing but the influence is clear to see. This is normally in the form of vibrations on the spacecraft that gives the appearance of a change in FSS pointing, though not in the longer term. The end result is that we never got the thrust that we expected on July 8th, whether or not we had indications of latch valve open.

Evaluation of July 8th Attempt and Plan for the 9th

Our evaluation of what happened that day is different than what we know now but it informed how we operated on the ninth and thus is instructive to deciphering the nature of the overall problem. Our evaluation that day was that we thought that the hydrazine tanks in fuel system 1 (on the left in figure 3) was depleted of fuel and pressure. Our pressure transducers indicated that there was no pressure in fuel system 1 and only minimal (about 4 psi) in fuel system 2. We also were very unsure of the telemetry indicator for the operation of the latch valves. Thus the plan was to first attempt to do the maneuver with the fuel system 1 (but use latch valve C rather than A) and thrusters E and M on the other side of the spacecraft. This would have the effect of using a different set of fuel lines but with the same fuel system tanks. Then we would then repeat the test using fuel system 2 and the same thrusters (E,M). If this did not work then we would open latch valve B on fuel system two and repeat with thrusters (E,M). If that did not work then the final test would be to use latch valve D with thrusters (F,N).

The Attempt on the 9th

The results on the 9th were pretty much the same as on the 8th. We initially made the same mistake by commanding latch valve C when the 28 volts was not on. The results were basically as they were the day before. Figure 7 shows the result of the first firing:

As you can see, the uppermost blue line is the accelerometer output. It looks pretty much like the trace in figure 4. It is opposite in sign because we used thrusters E,M, and sector 556 (the other side of the spacecraft) to start pulsing. The pale flesh covered lines are where we pulsed (21-30) the latch valve. There were other thruster firings done, but none of them with any effect. This ended the pass for July 9th.

Failure Investigation

After a short bout of team depression, we got started on our failure investigation. In learning how to be an engineer, it is just as instructive to study failure, and many times more so, than to study success. I have read virtually every failure investigation in the last 50 years for in space failures, from the early Ranger days to GEO comsats, and Shuttle failures. There is a common thread that not too many people write about, which is that these failure investigations almost always include people from outside of whatever organization was building or flying the system that failed, and that these guys were normally the best in the business. Another common denominator of such failure investigations it that they take time and lots of money. Our team has neither.

We seriously thought that there were no options going forward, but in order to understand what happened, and to see what we could learn, we decided to dig into the telemetry and see what happened and to see if we could determine the failure mechanism. No one on our team is an experienced hydrazine expert. My own expertise is more in communications, avionics, and power systems, with a good bit of experience in ion propulsion. Marco Collelouri, our controls and AOCS engineer is very smart, but without a lot of experience. Thus I felt, after receiving a few emails from people who offered suggestions on what might have happened, decided to throw the problem out to the world. I was astonished at the response.

The Collective…..

On the 10th we threw out a few questions related to some suggestions that had been sent to us by some of our global distributed network of supporters unsolicited. Keith put this on NASA Watch and http://www.spacecollege.org. We immediately started getting responses. While, as one might expect, many of them were uninformed, though enthusiastic, some were from the most qualified professionals in the world. I am not going to name names at this time and without their permission but literally, the very top tier of experts started weighing in.

We sent them telemetry, and information on the spacecraft. Fortunately there is a lot of information out there on ISEE-3, especially from the AIAA (see our references at the end of the article). With this, and with our telemetry that is also public, we started to build a picture of a set of plausible failure modes and the state of the system. I must stress that I have had multiple interactions with different groups of experts, some from industry, some from government, and some international. I have not shared a lot of the information from one group to the other in order to get their unbiased responses, until a consensus started to emerge. We also were able to be put in touch with (again, through a senior industry engineer), with some a retired person from TRW, the company that built the propulsion system in the first place. Even though this person did not personally build the one from ISEE-3 he did know the general design and had extensive experience with hydrazine propulsion systems.

The Conclusions of the Ad Hoc ISEE-3 Hydrazine Propulsion System Failure Study

After just a few days of consultations with these groups, and a consensus started to form some common elements stand out. Here is what we know about the state of the HPS system. Please refer back to all previous figures as this explanation unfolds.

Integrity of the Propulsion System Finding 1

It is quite clear from the telemetry that both fuel system 1 and 2 downstream of the latch valves was still fully pressurized. If this had not been the case, we would have not had a successful spin up maneuver, nor would we have had the initial thrust from both fuel systems on the 8th and the 9th. This means that the following hardware is working:

1. All thrusters have their seals intact and the thrusters provided impulse, showing that the catalyst beds are also intact (or at least mostly so).

2. The propellant lines downstream of the latch valves are also intact and were fully pressurized.

3. Temperatures in the system are high, in some cases, in excess of what is desirable, though not dangerously so. Temperatures in the propellant tanks are well within bounds.

Here is what we don’t know.

1. How much pressure is in the tanks (the options are failed sensors or depleted tanks).

2. The true status of the latch valves.

Procedures, Finding 2

We made a procedural mistake. That mistake was turning the latch valves on without + 28 volts applied. In our defense this was not in the AOCS procedure worksheet, though looking back into the old Mission Operations Plan, checking their status was part of the procedure. We thought that when we did not have an indication of operation, that the telemetry was dead, which is also the case in the communications system. Sometimes we imply the state of the spacecraft when telemetry is dead, by looking at whether the command that we sent was executed. So the latch valve status we thought was dead. It turns out this was incorrect. We corrected this in the middle of firing on the 9th, but it did not effect the propulsion system operation as can be seen in figure 7 and right at the right edge where we initiated propulsion but got no response. After we turned on the +28 volts and commanded the latch valves, they did successfully response (commands 21-30 as shown in figure 7). You can see that the latch valves vibrated the spacecraft in the accelerometer data. However, we did not have time before the end of that pass to fully investigate the issue.

Postulations

So the above is what we know. Now we had to use our own knowledge and that of our outside experts to eliminate, one by one, possible failures.

Propellant Tanks and Upstream Propellant Lines

In examining the telemetry we know that the HPS-1 and 2 fuel systems downstream of the latch valve were fully pressurized. This eliminates loss of fuel and nitrogen this way. We also know, and it was our rookie mistake, that nitrogen does not dissolve in Hydrazine, more than just by fractional amounts. This is why it is used as a pressurizing gas. We also know that this is a blow down system (see reference 2), which means that the nitrogen gas is mixed with the hydrazine and not in a separate bladder or pressure tanks. This is a simpler system and it eliminates failure modes. Thus there is only three ways that we could have lost fuel and pressurizant from the system.

The first way is through the latch valves. We already know that there was pressure downstream of the latch valves and thus that is eliminated.

The second way is that the fuel and pressurizant could have been lost through either the fill and drain valves or the fill and vent valves. There are two reasons this is not plausible. The first is that as far as our experts know, neither of these types of valves have ever failed in flight as they are physically capped before launch. Also, if they had failed, it is even less likely that they would fail over ten years after launch. Even less likely than that would those valves failed in both systems.

The third way would be a failure in the propellant lines upstream of the latch valves. While this has happened in the past. It is unlikely to have happened in both fuel systems, and if it had, there would be serious consequences for the spacecraft as Hydrazine eats spacecraft wiring and other hardware. If that had happened it is likely that the spacecraft would have been lost. Also, this would have had an effect on the attitude of the spacecraft, which we did not see when we first tabulated the telemetry after recovery.

Thus the conclusion of virtually all of our experts is that it is highly unlikely that fuel and propellant has been lost in the system. This brings us to the next postulation.

Latch Valves

The latch valves on this spacecraft were built by Hydraulic Research (see reference 2). These valves were popular and used on several spacecraft. In researching the company, we found documents related to pressure testing and leak testing of the valves. The valves do preferentially allow diffusion of nitrogen through the seals (we found this data on the NASA technical reports server). However, since the lines downstream were pressurized, it is unlikely that this happened. For this mechanism to operate would take further diffusion of the nitrogen through the thruster valves which is also unlikely with full pressure there.

What we did find, and I can’t be too specific here as this information is not in the public domain, is that there are different seal materials and that the type that most probably flew on this spacecraft is subject to temperature based swelling. Since we also see very high temperatures, in excess of 62 degrees C on the upper propellant lines, which are in contact with the valve, we now have a plausible culprit for our problem.

This possibility is enhanced by our own mistake in how we operated the latch valves prior to figuring out that we had to have the +28 volts on to actuate the valve. This is somewhat mitigated by the fact that later we did actuate the valves and did see physical vibration of the spacecraft from our opening and shutting the valve (commands 21-30 in figure 7 indicate that something happened). There is a case to be made that the valves did not actually open and we did not fully investigate this during our short Arecibo pass last week.

Temperature

The upper propellant lines near the latch valves is above their specified operating temperatures. We found that the line heaters have been on since the last propulsion maneuver in 1987, or over 27 years. While the temperatures are not dangerously high, the long term storage of hydrazine at elevated temperatures can cause the slow decomposition of hydrazine into ammonia and nitrogen and then eventually into nitrogen and hydrogen. Did this happen? We have no idea as there is no pressure transducers in the lines. However, one of our outside experts worked on the Magellan to Venus mission and gas evolution in the propellant lines was seen there.

Since this was a mission to Venus, a planet that gets twice the radiative heat that the Earth gets, and since ISEE-3 came considerably closer to the sun every 354 days for 27 years, it is very plausible that this happened. If it did, we would still get thrust out as we saw, but there would be gas in the propellant lines. We did see large drops in temperature in the hot upper propellant lines and large increases in the lower propellant lines. Rapid swings in temperature could be from gas and or hot ammonia (NH3 a decomposition product of Hydrazine) in the system.

Conclusion

Thus we have a plausible mechanism for our propulsion failures on the 8th and the 9th with the latch valves. High temperatures expanding the seal material could have either impeded the flow, or have precluded the latch valve from opening even with the microswitch indicated to telemetry that the valve was open. We also have high temperatures possibly evolving gas, causing a large gas bubble in the propellant lines Is this the case? There is a way to test both cases.

Corrective Action

There is a pretty good possibility now that we have pressure and or fuel in the tanks but that it is not getting to the propellant lines and out the thrusters. We are going to of course turn the +28 volts on this time! We will also open both valves on one of the fuel systems, the primary and redundant. We will also heat the tanks to see if we can see a rapid increase in temperature. If we see a rapid rise, that would indicate no fuel in the tanks (testing for all eventualities). There are several things we will do to test out and try propulsion to bleed all the gas out of the lines.

What we could see would be not much activity and then toward the end of the pulses from the thrusters we could see propellant flow, temperature increase, and thrust!

Cross your fingers. We will have a pass on July 16th at Arecibo, so we will soon find out what the outcome is.

The Collective Consciousness of the InterNet

There was a great article on space.io9.com related to distributed engineering and our project, and how the people from the net came together to help us. I first saw the term distributed engineering in the late 1980′s from the amateur radio community. It began through using ham radio to do this, then it migrated to email before the advent of web browsers, and then to the web. What happened with our call for help goes far beyond that as the distributed engineering meme begins with a pre organized group of people that collaborate in geographically disparate locations toward a common engineering goal. Before our call for help last week, I knew maybe one or two of the experts that came in and helped us. This goes well beyond distributed engineering to a collective consciousness. I often characterize the internet as the global extension of my brain, with vast stores of knowledge that the brain organizes through the interface of the browser.

In the beginning of the net we used this to research information. With the rise of the ubiquitous internet among the professional class and beyond in the world, we now have something never before seen on this scale in the history of mankind, a near instantaneous way to not only research information, but to rapidly organize humans to do “things”. We now have crowd funded efforts that bring people together of like interests to fund interesting projects like ours. We have crowdsourced collaboration in the arts, sciences, and engineering. There is a lot of talk about singularities in the technology world, and for the most part they are marketing myths from my experience. However, and this is what I leave the reader to ponder, we have reached a threshold where vast numbers of people can work together in a near real time manner to solve problems and do good and interesting (or evil) things. One wonders where this will go….

This is really great news, because a few days ago I thought this mission was now DOA.

My approach would be to first figure out a way to cool the local section of the spacecraft, so that the “O” rings aren’t swelled. And of course, turn on the 28 volts first.

If that doesn’t work, my idea if it is a stuck valve would be to “impact hammer it”, just like a jackhammer. Pulse it with a series of 28 volt spikes if your control systems allows it. I’m sure they already thought of this, but I’d be looking for valves in every space scrap yard there is and setting up a simulation in a heated chamber.

25 thoughts on “Breaking news on the ISEE-3 mission: it may not be lost – it's those "O" rings again”

In the big picture of the history of humanity this project isn’t much, but as an inspiration to what can be done by a few willing and talented people this is awesome. Even if this craft ends up as a terraform blister plunging into the sun some day, what has been done here is the stuff of dreams of at least my youth. There was a time when classrooms filled with inquisitive minds would sop this up like cream on a cat’s tongue. The shame is today someone somewhere is preaching to a room full of staring sedated skulls the huge waste of these space programs and how the money could have been better spent.

A text by a person that doesn’t seem to compreehend that was individuality that helped the mission, because several persons had several different experience and knowledge.
There wasn’t any collective. It was free people that had individual interest in helping to fix the situation.

Reblogged this on Sierra Foothill Commentary and commented:
I donated to his project and it was a downer when it looked like the gas tanks were empty. Now it looks like there is still a change, with a change in procedures and a little lick. The real message of this post is at the end, the Internet enables collective problem solving complex issues, bringing multiple minds with unique experience together to solve a technical problem. If you are not technical you can skip to the ending paragraph : The Collective Consciousness of the InterNet. For those interested to examine the progress is engineering and problem analysis read the whole thing.

AlexS says:
July 15, 2014 at 12:02 pm
A text by a person that doesn’t seem to compreehend that was individuality that helped the mission, because several persons had several different experience and knowledge.
There wasn’t any collective.

He did say “collective”, not hive which is what you have implied. The difference is a collective can include free thinking and free will but willingly work together to solve a common problem – even one none of them have. A hive is a DNA-based solution where classes of members are purpose-designed.

Hopefully someone provides the technical insight to make it work or a comment from someone ignorant of the technical aspects sparks an idea that leads to the solution.
Worst case, you’ve shown that not all the “space junk” out there is junk.

Still inspiring. Still on track. Without the drama the story, still to be told, would not inspire future generations as much as it certainly is going to.
First star on the right; press on till morning.

Great effort by Dennis and his team, kudo’s to all.
Anthony wrote; “My approach would be to first figure out a way to cool the local section of the spacecraft”
Sadly, the only viable way to do that would be to steer that section of the bird you want to cool away from Sun exposure, but we have no operating thrusters. And it’s spinning so just about every exterior surface is getting sun exposure. But good idea.
“It is quite clear from the telemetry that both fuel system 1 and 2 downstream of the latch valves was still fully pressurized.”
I’m sorry but respectfully I must disagree. I do not see a pressure transducer down stream of the latch valve (schematic is a bit fuzzy, I apologize if I missed it, I checked several times). The telemetry (accelerometers and FSS) are simply telling you that the spacecraft moved
My take on the telemetry is;
The first attempt activated the thruster valves
While very finely crafted, each thruster valve exerts a slightly different torque onto the spacecraft, they are not exactly balanced for torque. Just the original mounting tolerances (distance from the center axis of the spacecraft) are probably large enough to cause this. Since they are on the perimeter of the spacecraft one of them simply “twisted” the bird off course “to the left” just a little bit more than the other valve “twisted” the bird off course “to the right”. This imbalance is reflected in the FSS readings. Those are a little hard to read, but it looks like tenths of a degree.
The accelerometer is simply reflecting that something (the valve mechanism most likely) moved WRT the spacecraft.
My initial read of your telemetry is that the bird is “out of gas”, but I have not seen all the details you have so I am probably mistaken. I do hope I am.
Great project, thanks for all the updates, and keep plugging.
Cheers, KevinK

Whoops, just one clarification, simply activating the thruster valves (with no gas pressure present) could cause a torque imbalance. Each has a mass (magnetically active) that is moved when electrical current is applied to the coil inside. Everybody’s heard the “clunk” when the water valve inside your washing machine calls for water right ? Some of that sound is the solenoid core (usually a steel rod 1/2″ diameter by 1″ long or so) slamming into the “end of travel” part of the valve mechanism. That force is present if water is in the hoses, or not.
If you put accelerometers on your washing machine (ok, only a crazy engineer would do that) you should be able to measure the “valve activation torque” when no water is present.

“in a way that few philosophers or technologists have envisioned, we are connected in a way never before thought possible.”
Unless you went to high school, in which case you would have been given Forster’s story “The Machine Stops” as part of your English course.http://archive.ncsa.illinois.edu/prajlich/forster.html

We often take it for granted here on WUWT but we see that collective intelligence applied here every day in various threads where individuals with unique personal experiences add to the discussion by relating information that the author of the thread did not even know existed, let alone had searched for.
In technical areas like IT you also see it every day in trouble shooting system and software failures. It is a great resource if people are smart enough to enable those random bits of information to come together to solve a problem.
Great news to hear that the space craft may be viable after all.
Like the fascination we all felt at the time of the early space flights and how the Apollo 13 incident focused the attention of millions of individuals on a single drama and its solution. I hope that this project serves to help inspire another generation to reach for the stars and do things that people think are not possible.

Unless you went to high school, in which case you would have been given Forster’s story “The Machine Stops” as part of your English course.
We read Shakespeare, Melville, Shelly, Chaucer, and Jonathan Swift, as well as many more. Science fiction was low on the list.
I did get to visit the U.S. Space and Rocket Center and get a tour of the inside of Skylab when the mission was going on. That has to suffice..

“We read Shakespeare, Melville, Shelly, Chaucer, and Jonathan Swift, as well as many more. Science fiction was low on the list.”
Chaucer, Shakespeare, Milton, Dryden. Pope, Keats, Tennyson, Donne, far too much Thomas Hardy, likewise Wordsworth (bores, the pair of them) Slessor, and more. Forster was the only bit of sci-fi (in an anthology of short stories) but he was there. Of course, we read lots of more modern sci-fi outside school. (Kids could and did read in those far-off days.)

“He did say “collective”, not hive which is what you have implied. The difference is a collective can include free thinking and free will but willingly work together to solve a common problem – even one none of them have. A hive is a DNA-based solution where classes of members are purpose-designed.”
He praised the borg which he specifically said : All of the borg are connected to each other through a communications link that allows each of them to share each others thoughts, though in a manner that erases individuality. He then talks about “collective consciousness” in last part.

Another superb update on this imaginative undertaking.
One minor point I would like to mention is that although Dr Feynman picked up on the O Ring issue that caused the loss of the Challenger Shuttle, the real source of that analysis was Allan McDonald of Morton Thiokol. His account ‘Truth, Lies and O Rings is well worth reading.

So, we know that the latch valves permit the preferential leakage of nitrogen gas, and we’ve also seen that it’s quite likely the hydrazine fuel has been decomposing into nitrogen and ammonia lo these 27 years, especially since the line heaters have been constantly on. Thus it would seem that the fuel tanks now contain a “rich” mixture of ammonia and hydrazine and that the mass of the remaining fuel is asymmetrically distributed in slushy agglomerations throughout the fuel system. The upshot here being that the spacecraft is out of balance and is burning the equivalent of damp wood. This would explain why the spin-up maneuver was successful (since there is still enough decomposable hydrazine to provide some thrust), and would also explain the anomalous change in the sun angle (internal redistribution of asymmetric mass). The best chance for a successful course correction would come after raising the temperature and allowing the spinning spacecraft to adequately mix the remaining, substandard fuel (if there is any).
Just a brainstorm.

We use cookies to ensure that we give you the best experience on WUWT. If you continue to use this site we will assume that you are happy with it. This notice is required by recently enacted EU GDPR rules, and since WUWT is a globally read website, we need to keep the bureaucrats off our case!OkPrivacy policy