16 years and counting: How ESA squeezed oodles of bonus science out of plucky Mars Express probe

MEX I can: A tale of ingenuity and innovation

Special report As the European Space Agency flicked the standby switch on some of its long-lived spacecraft in response to the COVID-19 outbreak, The Register figured it was time for a look at how the agency has kept its fleet flying far beyond expectations. Today, the veteran Mars Express orbiter.

ESA's Mars Express (MEX) spacecraft is an interesting beast. Launched on 2 June 2003 from Baikonur aboard a Soyuz launcher, the 1,120kg vehicle (including the ill-fated Beagle 2 lander) arrived at the Red Planet on 25 December 2003. The Beagle 2 lander aside, the nominal mission plan for Mars Express (or MEX) called for the spacecraft to spend one Martian year (about 687 days) in orbit. Nearly 17 years later, MEX is still making discoveries.

We spoke to James Godfrey and Simon Wood, part of the team responsible for operating the spacecraft, and project scientist Dmitriy Titov about the challenges and benefits of keeping MEX ticking over long after the lights were expected to have gone out.

Keeping the lights on

The MEX story is one of overcoming potential mission-enders and finding oneself with an improved spacecraft at the end of it. The thing even "grew" an extra instrument over the course of the mission. "Innovation out of desperation" is how one might put it, although ESA would probably prefer "from challenging situations arises opportunity".

The first such challenge came early on as engineers discovered that a wiring error had left MEX with only 70 per cent of the expected power. "Originally," Godfrey and Wood told us, "it was as low as 60 per cent, but we clawed back another 10 by tweaking the fine-tuning of part of the power subsystem."

While margin built into the spacecraft helped, the duo were swift to heap praise on the spacecraft manufacturer, which updated the software to enable the flight control team to better manage the power and, importantly, the batteries.

"The whole mission concept was adjusted as well," the duo told us, "the original idea of having transmitters on most of the time, payload all the time had to change. And we had to come up with a new way of flying it, which involves very careful management of what was turned on when." And, of course, taking very good care of those batteries.

Memory corruptions

Beagle 2 and a glitch with a Mars Express Sub-Surface Sounding Radar Altimeter (MARSIS) boom (the latter rectified by a bit of solar heating) aside, it wasn't until 2011 (six years after the nominal two-year mission) that the gang encountered a potentially terminal situation – safe modes triggered by corruptions of the spacecraft's command timeline.

As originally envisaged, the commanding of the spacecraft was performed by a mission timeline of 3,000 commands, which required constant topping up from the ground (or "feeding the monster" as one former operations manager called it). That timeline sat in the Solid-State Mass Memory (SSMM) system and by the end of 2011 it was clear that all was not well.

The issues didn't prevent the SSMM from working but, critically, "it meant we couldn't rely on it".

"We couldn't rely on the communications between the mass memory and the flight computer," recalled the team. An occasional corruption of science data wasn't the end of the world, but corruption of the commanding timeline, also stored in the SSMM, would be "a very, very bad idea".

The spacecraft would protect itself from such a corruption by triggering a safe mode, which is all well and good except that analysis showed that the team would wind up with two safe modes a month. With each safe mode (with safing and recovery) consuming four months' worth of the annual fuel budget, "we'd have emptied the fuel tanks pretty quickly if we hadn't done something."

While MEX uses reaction wheels (more on those later) for its attitude, the end of the fuel is pretty much the end of the mission.

Although originally suspended for six months, the team were able to get limited operations running within a month thanks to another timeline, intended only for emergencies, lurking in the RAM of the onboard computer and only 117 commands long.

It then occurred to the team that "what we can do is put all the commands as files in the mass memory and then load them on a just in time basis into this shorter memory as required."

Stir in some safeties, such as not executing any commands if the whole file was not loaded and making sure that each file was fail-safe and the gang had themselves an improved method of managing the mission.

As well as getting a basic version up and running within a month and resuming normal operations in six, the team found that "the file based operations are a much better way of operating the spacecraft, it means we can load all of the files for a week or longer than a week in one uplink which greatly simplifies operations."

Rather than having to continually feed the beast, "now we need about an hour-and-a-half, maybe two hours, a week. We send all the commands for the following week. If for some reason we need to cancel an operation we can just delete the file or we can actually disable its operation."

Hindsight is a wonderful thing, and an observer would be forgiven for wondering why things weren't done that way from the get-go. It is, as the team agrees, "so much simpler, so much more flexible".

The reason is a simple one – Mars Express was ESA's first mission to orbit another planet. "We'd never actually operated a spacecraft like that before," the duo said, so it had been designed around agency experience of "how it should work".

Going without gyros

The next potential mission-ending issue came in 2017 as the gyros required to control the spacecraft's attitude showed signs that the end was approaching.

MEX has two Inertial Management Units onboard (IMU), each containing three gyros. Attitude data from these devices is used by the flight computers when spinning up MEX's reaction wheels in order to point the spacecraft.

While industry support for the gyros had ended in 2009, the team had inadvertently bought itself a little more time thanks to the lower-than-expected power availability. The lack of power meant that the heaters had been turned down and the spacecraft operated colder than planned. The team reckon that a side effect of this was that the IMUs actually aged slower and outlasted those of Rosetta.

But at least four of the six gyros were on their way out. The lifetime is measured by the Laser Intensity Monitor (LIM) current drawn and the original manufacturer had provided a graph showing how that current would evolve over time.

"It's rather a complicated graph," the team explained, "and it isn't until the gyros are getting quite close to the end of their life, it's difficult to know that they are where they are."

And so, in the early part of 2017, the MEX team saw that four gyros had entered that final phase and the mission could be over within two years.

"Which was extremely depressing."

However, the spacecraft still had its Star Trackers that could potentially be used to measure its position and thus reduce the load on the gyros. ESA's comet mission, Rosetta, had just such a mode (albeit a very simple one) so could the team lift the Rosetta code and drop it into MEX?

Rewriting the onboard software (and enhancing the Rosetta code for MEX's needs) was possible, but getting the manufacturer to do so would be costly – after all, support had run out in 2009. Even if it could be done, the team would need to recharacterise the dynamics of the spacecraft when running in "Rosetta Mode" and the clock was ticking.

Ruefully, the team decided to see how they could maximise the science return from the time MEX had left.

"And that's how we left it for a few days."

Except they didn't. The Flight Dynamics team took a closer look at the problem and popped back up with "you know that crazy idea of rewriting the software? Well, maybe it's not so crazy…"

The flight software for MEX looked a lot like the Rosetta version, to the point where "we could see from looking at the code you can see where the gyro system fitted into the software…"

A meeting was held – could the team simply copy and paste the Rosetta code into MEX? Kind of, but since the Rosetta mode was designed for the cruise phase of the probe, it didn't immediately bring too much benefit; maybe an extra year. MEX needed a gyro-less mode that could operate not only for Earth pointing, but also during slews and science pointings, and it needed to operate for 80 to 90 per cent of the time. The team simply had no idea how to do it.

"It was a very, very depressing meeting."

But that 80-90 per cent figure was enticing. Getting the gyro duty cycle down to 10 per cent would buy MEX another 10 years of science. While Rosetta's basic algorithm was a good starting point, MEX needed something more flexible. The team also had the source code for both, the software development environment and a compiler.

But as Godrey and Wood observed: "This is not something that we would routinely do." Usually, industry would deliver a patch and the MEX team would check it and then upload it to the spacecraft. But actually going elbow-deep into the source themselves, compile and install it was a leap into the unknown.

The thinking was that it couldn't possibly work but at that point, what did the team have to lose? The project was bound to be derailed by some sort of showstopper on the ground before it went anywhere near the spacecraft. So the morning after the meeting it was decided: "How about we have a go at rewriting the flight software?"

Ultimately, it was just an engineering problem that could be broken down into smaller pieces: solve A to get to B, solve B to get to C and so on. The engineers just kept moving forward until they hit something that could not be solved.

"I was continually expecting us to find the showstopper," said Godfrey, "and we never did."

Testing was tricky – there was no duplicate of the hardware on the ground for testing the code and support had ended in 2009. There was a validation facility for the Rosetta probe (sitting on a Sparc box), but it had never been configured for MEX and the clock was ticking for those gyros.

What they did have was a spacecraft simulator, which included an emulator of the onboard processor. It would have to do.

While support had officially ended, ESA did set up a small contract with Airbus (who had acquired the original spacecraft manufacturer) in a role reversal that saw Airbus review the MEX team's work rather than the other way around.

As it transpired, Airbus turned out to be hugely helpful, blowing dust off some testing tools, making suggestions and, effectively, "putting the band back together" as those who originally designed and built MEX (and had not yet shuffled off into retirement) were hauled back to offer assistance.

The first glimmer of "we might actually pull this off" came when the test version, replete with Rosetta mode, first fired up on the simulator.

MEX was not out of the woods yet. The only input into the system would be coming from the Star Trackers and while Rosetta winding up in safe mode if the trackers lost track would not be the end of the world, repeated safe modes for MEX would be bad due to the cost in terms of fuel.

Safeties were therefore built in – both Star Trackers were to be used at the same time, so if one lost track, the other could be used. The gyros could also be fired up automatically if there was a problem. Additionally, the file-based operations needed changing to bring up the gyros if a file failed to load.

Finally, the team had to work out how to get the code, compiled from ADA-83 source, into the spacecraft. The resulting binary was broken up into individual commands to write to the spacecraft's EPROMs, with the backup computer receiving attention first, followed by the prime computer.

The full EPROM image took approximately an hour-and-a-half to transmit before MEX could be rebooted with the new software. The process was not as risky as it might seem, since the first reboot would bring up the backup computer and if something had gone wrong, a further reboot would switch back to the primary that had yet to be touched. MEX also has a basic version of the flight software burned into it which the spacecraft would eventually boot back to if both computers were unable to load anything at all.

"Breaking the spacecraft would be a bad idea," the team said.

Indeed, it was six months after a successful boot of the backup that the gang decided to update the primary.

As it transpired, it was precisely one year and one week after that original meeting that the new methodology was adopted and the duty cycle of the gyros dropped to 10 per cent. One gyro has since failed (on 26 August 2019 – over a year later than expected) and five remain. At worst, the gang reckons they will still have at least three working until 2026, although plans are afoot to push that date out further.

Longevity

All good things must come to an end, although what will actually kill off MEX is open to debate.

The spacecraft is very much running on empty. Fuel usage is tricky to calculate and the MEX team reckons there is around 6kg of fuel left. Unfortunately, that figure is smaller than the uncertainties used in the calculation. The gang has looked to the example of a similar spacecraft, Venus Express, and based on that experience foresee fuel lasting until 2030.

The gyros remain a worry since they are required when the spacecraft does two hours' worth of momentum dumps per day. Cutting the time they are used during the dumps, or ditching them altogether could buy a few more years – again until around 2030.

Finally, there are the lithium-ion batteries for which the gang had some good news from Airbus. Although well past their expiration date, the batteries have turned out to be far less degraded than expected and could also make it to 2030.

Growing extra instruments

The sheer longevity of MEX, coupled with the reliability of its instruments, has meant scientists have been able to gather far greater amounts of data on the Martian environment than the original mission could have hoped for.

That data includes information gathered from a bonus instrument after the Visual Monitoring Camera (VMC) used to observe the departure of Beagle 2 was switched back on in 2007 after four years powered down. A young (at the time) engineer had wondered if the camera could be brought back online and, sure enough, it fired up and was able to send back images of Mars. Not to the same resolution as other orbiters, but "as a bit of fun", according to the MEX team. It was handy for outreach and a Flickr account was set up.

The observations used were Earthbound because "there was a little section in their paper about stating that they couldn't use observations from the orbiting spacecraft because none of them could get the entirety of Mars in one shot in the cameras because of the orbits they were in."

We would like to continue observations as long as possible to see how the system behaves in response to different kinds of impact, different kinds of phenomena in the surrounding space and on the Sun..

It transpired that the authors were unaware of MEX's archive of images with the whole of Mars in one shot.

"They're effectively now handling the science operations with the camera," explained the team. "They've had a few papers published already: they've done analysis of the Martian dust storm seasons. They're looking at the polar vortex, which is being observed for the first time. The cloud that was over was over a volcano that people thought was erupting that was very nicely observed using VMC as well."

Further bonus science thanks to the equipment bolted onto MEX for Beagle 2 is also hoped for, in the form of the VHF radio intended for communications with the unfortunate lander. The radio has been used for communication with various NASA spacecraft as well ESA's own Schiaparelli mission and investigations are under way into using the device in conjunction with ESA Trace Gas Orbiter (TGO) to perform occultation radio science.

Project scientist Dmitriy Titov added that as well as spacecraft-to-spacecraft radio occultation (hopefully to be tested this year) the extra time had allowed scientists to use the available payload "in different operational modes".

He cited the example of the sub-surface radar, which had been used in what he described as "low-resolution mode" to ensure a survey of the planet could be conducted during the nominal mission. Onboard processing was used in order to moderate the data volume, according to Titov.

However, the mission extensions allowed scientists to go further, allowing more "high-resolution passes", bypassing the onboard processing and sending all the data. "It gives much better resolution of different tiny features," he recalled. The subsequent discovery of subsurface pockets of water, he added, "was not possible with the old fashioned standard mode of [the] radar. But when they switched to the high-resolution mode, they managed to see tiny details."

Though there has been some inevitable degradation in some instruments over the years (Titov cited the loss of one of the channels of MEX's OMEGA instrument some 10 years ago) the payload continues to operate and scientists remain keen to squeeze every last bit out of the mission: "We would like to continue observations as long as possible to see how the system behaves in response to different kinds of impact, different kinds of phenomena in the surrounding space and on the Sun."

Indeed, having observed the planet for one complete solar cycle, MEX is now entering the second, giving scientists a chance to observe and extrapolate the history of the planet.

Paying tribute to the original spacecraft designers for building in the margins that has allowed MEX to operate well past its expected mission and potentially into a third decade, the team told us: "All of us feel very lucky to do the job we do.