More details on the delay of Mars Science Laboratory

It was a lengthy press briefing today, with the four bigwigs on the press panel being exceptionally long-winded. Here's the short version, which I'll follow up with more detail.

Unanticipated problems, principally with the motors and gearboxes (collectively called "actuators") needed to drive every moving part of the rover, have resulted in there simply not being enough time to prepare MSL for the planned October 2009 launch period.

In particular, life cycle testing has not even begun for the actuators yet. Since MSL is planned to be a very long-lived mission, this is an unacceptable situation.

A few-month delay would (probably) have been sufficient, but because Mars launch opportunities occur only about every 26 months, the only option was to delay to 2011.

The added two years will be used to solve problems and to do additional testing; no substantive modifications to payload capabilities (i.e. adding instruments) will be permitted. The time will also be beneficial to incorporate lessons learned from the difficulties with sample handling experienced by Phoenix.

The additional cost of the two-year delay is $300 to 400 million, resulting in a total life cycle cost of $2.2 to 2.3 billion.

They believe the cost can be absorbed without canceling any planned missions, but "a delay to a major planetary mission" is likely. They will look within the Mars program for money first and then widen it to the rest of the planetary program.

Looking beyond MSL to future Mars exploration, particularly Mars sample return, Ed Weiler announced a handshake agreement with his counterpart at ESA, David Southwood, to work together so that the future of Mars exploration will be a joint program between NASA and ESA.

The four people on the panel today were Mike Griffin, NASA Administrator (that's the head of NASA, the one person at NASA appointed by the President); Ed Weiler, the Associate Administrator for NASA's Science Mission Directorate (who recently replaced Alan Stern, who has been much in the news on the topic of MSL budget overruns of late), Doug McCuistion, Director of the Mars Exploration Program at NASA (he's the one at NASA who has most direct oversight of MSL and is thus most knowledgeable about the specifics of the problems), and Charles Elachi, the Director of the Jet Propulsion Laboratory, which is building MSL for NASA.

Model of MSL's radioisotope thermal generatorThis device is a model of the radioisotope thermal generator, or RTG, that will power the MSL rover. The model does not contain any plutonium, but it is capable of generating the huge amount of heat that the spacecraft must somehow dissipate safely while waiting for launch and during the cruise to Mars. The model was waiting to be installed onto the flight rover in advance of thermal testing. Credit: Emily Lakdawalla

MSL under construction

As of November 17, 2008, the partially-built MSL rover had been stacked with its descent stage, then encased inside its aeroshell (white conical shape), and stacked with its cruise stage (above the aeroshell), in preparation for thermal vacuum testing. The gray skin stretched over the bottom of the aeroshell simulates the heat shield, which had not yet been delivered. An engineer in the middle ground at right provides scale. The open hatch in the aeroshell provides access to the rover for workers to install the nuclear power supply, which will not be installed until just days before launch, because of the radioactive hazard it represents to human workers and because of the amount of heat it throws off, which will have to be dealt with aggressively by the MSL's cruise stage using active cooling systems. The red-and-black-capped round object on the cruise stage that is facing the camera is a navigational device (a sun sensor or star scanner) that will be used to help steer the stacked spacecraft to Mars. MSL will be stacked and destacked numerous times as it is tested in preparation for its 2011 launch.

They reported that the cruise stage (the spacecraft that will steer the spacecraft from Earth to Mars, handling the navigation and Earth communications along the way) is nearly complete. The descent stage (which will lower MSL to the ground) is basically complete. The rover is mostly built, with five of its instruments delivered, and the rest scheduled for delivery in January. But the fact that the actuators have not yet been delivered is a big, big, big problem. McCuistion said that the rover is designed to contain a total of 31 of these actuators in a variety of sizes. That means they need to build 60 flight-qualified units (basically, two of each; after testing, the best one will be placed into the rover), and then there will be 45 provided as engineering models, which are used for life-cycle testing and for mission operations, when they will need to troubleshoot any problems they encounter on Mars with duplicate copies of the actuators on Earth.

There are great big ones that will drive and brake the wheels and rotate them on the ends of the legs. There are good-sized ones in the robotic arm, in shoulder, elbow, and wrist joints. And there are numerous small ones within the mechanisms designed to collect and deliver samples to the package of analysis instruments within the belly of the rover. Not a single flight actuator has yet been delivered to JPL; they keep encountering problems. McCuistion said they've encountered "workmanship problems," mentioning bearings not being installed exactly correctly, and problems with devices within the actuators that are designed to measure how much they've moved. Currently they have a new problem that "we don't really understand the cause" of -- something having to do with the motor braking mechanism. "We have a drag torque problem and a brake release problem that we don't understand," he said.

Until they work out these problems, they can't even begin life cycle testing, where they take engineering models of the actuators and run them and run them and run them, simulating two years' worth of operations on Mars, to see if the motors really can last throughout the planned mission. My understanding of the situation is that they would have enough time between now and October 2009 to do these tests -- but with almost no schedule margin (I've heard 10 to 20 days is all that remains) to fix any problems that are encountered. That's risky. The only way to make that risk go away is to delay, and once you delay past October, celestial mechanics dictates a 26-month delay.

Now, looking only at the MSL mission, a delay will be beneficial. They were really racing to meet the October 2009 deadline. Two more years will mean that much more extensive testing will be performed, helping them to weed out any more unforeseen problems. In particular, two more years will allow them to really take a good look at the experience that the Phoenix mission had with handling real Mars samples, and see if any modifications are needed to deal with some of the difficulties that the Phoenix team encountered. The first formal meeting between the Phoenix team and the MSL engineers only happened in October of this year. Already, based upon that meeting, they are adding a device to the exterior of the rover that can be used to clean the sample collecting head from time to time. (Adding this device was one reason they removed the sample cache from MSL -- they needed the space on the outside of the rover in an area reachable by the arm for this cleaning device.) The cleaning device has no moving parts, so it wasn't too complicated to add, and didn't have any responsiblity for the launch delay. Now though I wonder if they'll have more time to test the MSL sample collection mechanism using simulated Mars materials that have properties similar to what Phoenix encountered -- and to modify the design if they encounter similar difficulty.

But MSL doesn't exist in a vacuum, and the additional $400 million has to come from somewhere. But figuring out where is not straightforward, because the $400 million doesn't all come in one year. The good news is that MSL was to cost an additional $200 million in the 2009 fiscal year to prep it for launch on time; that $200 is replaced by the $400 figure, so it's not a total $600 million over. There will of course be money spent in the rest of 2009, but it shouldn't be much more than had previously been budgeted. The bulk of the additional $400 million must be spent in 2010 and 2011, when MSL is being tested and prepared for launch; then there's a small additional amount for operations in 2013 and 2014, when NASA hadn't originally planned on spending money supporting MSL.

So the $64,000 question -- or, more accurately, the $400 million question -- is, where can NASA get hundreds of millions of dollars in 2010 and 2011? The following is purely thinking out loud and speculation on my part, not informed by any official word. They say they'll go to the Mars program first, but I'm not aware that any Mars mission will be spending that kinds of money in 2010 and 2011. Presumably Mars Reconnaissance Orbiter will still be operating, but they certainly can't raid that too much, because it'll be MSL's telecommunications link to Earth once it arrives. Maybe Odyssey will still be going, but its operational budget has got to be tiny by now. May 2010 will see the next winter solstice in the southern hemisphere -- as much as I love the rovers it's hard for me to imagine that Spirit can survive one more winter; possibly Opportunity can, but it won't be costing much. There's only one Mars mission under development, the next Scout mission, MAVEN; it's not scheduled for launch though until 2013, so I would assume it doesn't start spending its $400 million budget until closer to 2012 and 2013, so although a lot of people look at MAVEN as one $400 million chunk that could pay for everything on MSL, that money isn't showing up in the right time frame. Delaying it to 2016 probably would only net you a few tens of millions, maybe a hundred million, for MSL, I'm guessing (and that's just my guess, I don't know for sure).

So we look outside the Mars program. Looking at the calendar, what else is under development in 2010 and 2011, launching in one of those years? There is only one planetary mission that I see launching in that time frame: Juno, which is scheduled to launch to Jupiter in August 2011. Juno would be spending hundreds of millions of dollars in 2010 and 2011. So, that's my prediction. I think we're going to be seeing NASA announcing various token cost cuts here and there in the Mars program, but that they have to delay Juno in order to pay for MSL.

Many of the media questions focused on why cost overruns always happen. If you disregard Alan Stern's recent claims in opinion pages that MSL was originally budgeted at $650 million and take the NASA figure of $1.63 billion, $2.3 billion represents a more than 40% cost increase. They answered that they can only estimate costs accurately when they understand what is involved in developing a mission. So they could easily get an accurate cost estimate to duplicate the Mars Exploration Rovers, for instance. But NASA is not in the business of duplicating past missions; in each mission, they try to do things that have never been done before. Under such circumstances, it's impossible to foresee all the difficulties that will be encountered, and each time something happens, to fix the problem they have to spend more money.

So, is the answer to simply allot more financial reserves to every mission? Mike Griffin explained that it isn't, for numerous reasons. First of all, to have more reserves available just creates a self-fulfilling prophecy, he said. He explained that when they do cost estimations, they "fill in everything they know," and to inflate the budget figure at that point means "you're just making it up." He also said that the federal government -- the Office of Management and Budget and Congress -- "do not like big reserves. They represent money on the table that is not being utilized in the furtherance of other goals. As long as it's sitting in the reserve bucket it's not being applied to other national goals. Their view is, 'if you need more money, come back and tell me you need more money, and tell me why; I'm not going to give you more money in advance.' And I'm not sure that's wrong."

When you try to do something that's never been done before, there's an element of risk. My husband's an economist, so I know that in the business world, when a company faces risks, it tries to insure against those risks. There are various ways to insure. For example, you can take out actual insurance policies; or you can diversify your activities so that any one risky thing can't have too great an effect on your whole bottom line. There isn't any independent party that can insure the federal government for multi-billion dollar projects though. In essence, the government is self-insured -- the way it handles risk is to be aware that the risk exists, and to be prepared to have to deal with the financial implications of a cost overrun in one part of their activities by cutting activities elsewhere. What we're seeing now is the beginning of that self-insurance process. NASA says that the situation is not so bad that they'll have to cancel a mission -- but they may have to "delay ... a major planetary mission," Weiler said today, which, again, I think points to Juno.

I tried to ask two questions at the briefing. My first question had to do with the power supply, which depends upon the decay of radioactive plutonium. Over the two-year delay, the plutonium that has been alotted to MSL will be steadily decaying. How much would that be, I asked, particularly since, as far as I understand it, MSL's daily driving distances will be power-limited? The answer was that the delay would be about five percent, which they did not regard as significant. Somebody has since told me that the limitation on drives doesn't have to do with the amount of power that the RTG can generate but instead with the amount of charge that the batteries can store.

The other question I wasn't given a chance to ask. MSL, like all the other Mars landers, depends on an orbiter for the bulk of its communications with Earth. I imagine that, like Spirit and Opportunity, MSL will have direct-to-Earth capability; the Mars Exploration Rovers use this for commanding, but it's not high-bandwidth enough to send down all the data that they gather. Odyssey has been in orbit for seven years already, and though it may still be around in 2012, it's not possible to rely on that. Mars Express has been there for five years, but although cooperation between ESA and NASA is good, I don't think we can reasonably expect an ESA spacecraft to provide daily support to a NASA mission, and anyway we don't know if Mars Express will last that long either. Mars Reconnaissance Orbiter, then, is the only orbiter that will likely be available for the lion's share of MSL data relay. By the time MSL does arrive, MRO will have been in orbit for six years. I don't see any reason why MRO won't still be operating after six years, but MSL is supposed to last for at least two more, and hopefully several more beyond that in extended missions. Isn't it risky to assume that MRO will be there for such a long time? There's no future telecom orbiter in development -- there was once such a thing as a Mars Telecommunications Orbiter in development, but it was canceled a while ago. What happens to MSL if MRO dies? This seems like a single-point failure possibility, and I don't have an answer to that question.

UPDATE at 1:36 pm my time: I did get an answer from Dwayne Brown, the NASA SMD public affairs officer, on the comm question. He wrote: "The orbiting assets that are expected to be available to support MSL operations when it arrives at Mars (on the revised schedule) include MRO, Mars Express, and Mars Odyssey. Although two years later, it is still expected that these orbiters will be in place and available for communications relay support for MSL. Having multiple relay-capable orbiters in place allows the continuation of MSL support if one (or even two) of them should become inoperable. In the very unlikely event that all three orbiters should become unable to provide comm relay support, the MSL rover still has the ability to utilize direct-to-Earth (DTE) communications. Albeit slower, the MSL mission could be completed utilizing the DTE link only."

Moving beyond MSL, there's the good news about the coming NASA-ESA collaboration on Mars exploration. That's such good news, and so significant, that I think I will cover it in a separate blog entry.