Imaging Masterminds Give Human-Scale View of Mars

July 11, 2004

The world gets to see the rusty, dusty martian terrain as if humans themselves were riding atop the rovers. Who’s responsible for our front-row seats on Mars? The masterminds working tirelessly behind the scenes in the Multimission Image Processing Lab (MIPL) at the Jet Propulsion Laboratory. Like professional basketball stars, MIPL (pronounced ‘mipple’) team members at Jet Propulsion Laboratory perform amazing technical feats and shoot images from Mars into the Internet in record, near real-time before their shot clocks expire many times a day.

The world gets to see the rusty, dusty martian terrain as if humans themselves were riding atop the rovers. Who’s responsible for our front-row seats on Mars? The masterminds working tirelessly behind the scenes in the Multimission Image Processing Lab (MIPL) at the Jet Propulsion Laboratory.

Like professional basketball stars, MIPL (pronounced ‘mipple’) team members at Jet Propulsion Laboratory perform amazing technical feats and shoot images from Mars into the Internet in record, near real-time before their shot clocks expire many times a day.

Not only do they create the seamless landscape pictures from Mars, but the imaging team also keeps the science game moving by creating image products that enable the rovers to rove and the scientists to collect science.

”We’re what’s under the hood,” explained Helen Mortensen, former MIPL team leader and development engineer. ”You won’t see us at press conferences, but we’re behind the creation of the images and the safe movement of the rovers.”

The Invention of Digital Cameras and Image Processing

”For digital imagery, MIPL is where it all started in the 1960s,” explained Payam Zamani, MIPL design and system engineer.

In the early days of space exploration, engineers had to figure out a way to take a basic television signal and convert it to zeros and ones in order to store image information onto a computer and beam the pictures back to Earth.

Thus, digital cameras and digital image processing were invented at JPL for the space program, and enabled the first views of the moon and Mars.

We’ve certainly come a long way in a few decades. Not only do we have better and better pictures of Mars and the other planets we’re interested in exploring, but we also have seen a number of benefits here on Earth. MIPL is an unsung hero of sorts.

”The technology designed for the space program has now made it into the commercial world, and anyone who uses a digital camera and a home computer to download and process pictures is using MIPL technology,” said Zamani.

Scissors, Paper, and Paste for Space-Age Images?

”When the first photos were transmitted from space, scientists would have to use scissors to cut and paste multiple images together to make a panorama, and it would take days or months to build a picture of a large landscape from another planet,” said Mortensen.

Now, utilizing software that JPL has been developing and improving for over 30 years, scientists and engineers can see images from Mars within 60 seconds from the time the data hits Earth.

MIPL processes 30 to 250 images a day, per rover. Through a MIPL software program named File Exchange Interface, museum partners around the nation can also get high-resolution images from Mars out to the public in nearly real time.

More than 100 museums were able to participate, bringing the excitement of Mars exploration to people in the communities where they live. MIPL team members were even able to visit some of the museums, talking with students about their work.

Three-Dimensional Terrain Maps and Reaching the Rocks

A picture of a beautiful landscape speaks a thousand words, and shows what MIPL team members pull off on a daily basis.

However, other important but less obvious products generated by MIPL are the behind-the-scenes, high-tech stereo images that are used by rover drivers to maneuver safely on Mars.

Without knowing the distance to each rock near the rover or slope in the terrain around the rover, rover drivers could not safely command the rover to drive toward a specific location or send the robotic arm out to reach a target with the scientific instruments.

MIPL image processing uses some basic geometry to find distances of objects from the rovers.

”Geometry was my favorite subject in high school, but back then I didn’t know why those techniques were taught, and now it’s good to see that stuff teachers tell you to do actually can be used on exciting things like space missions,” said Zamani.

Working just like human eyes, the MIPL team uses two images taken from different angles — a left and a right — to determine depth in the images from Mars.

”If you hold your finger in front of you and close one eye, you’ll see that your finger moves if you look at it using your left eye or your right eye. That’s parallax.

Now move your finger out far from your eyes and look at it from your left eye versus your right. Things that are closer jump more than things that are far away,” explained Zamani.

The MIPL team finds common features in every left and right image pair taken from the same rover position, so they can calculate a triangle and figure out the distance of the common object from the rover, making a three-dimensional map.

Astoundingly, MIPL software does that pixel by pixel for 1,000,000 pixels per picture pair.

”There are very heavy-duty math algorithms in the MIPL correlation software that find matching pixels in the right image for every pixel in the left image,” said Bob Deen, chief software developer for the MIPL Mars team.

Know Your X,Y, Zs to Move on Mars

MIPL’s 3D maps enable the rover drivers to command Spirit and Opportunity from over 100 million miles away.

The IDD (Instrument Deployment Device) maps and range maps determine which points are reachable by the arm’s various instruments.

The XYZ maps and terrain wedges represent the 3D terrain to ensure the rover won’t hit anything, and the slope images show slope changes that could be hazardous to drive down or up, Deen explained.

The MIPL team creates these maps from telemetry that comes down from each rover multiple times a day.

They organize the ”metadata” for every image, which is over three hundred lines of essential information per image, such as exposure time, temperature, and rover body orientation and articulation at the moment the image was taken.

The team must process all of this information and whip out 14 vital products per image at a breakneck speed.

That way, scientists and engineers have enough time to digest the information and utilize the maps to command the rover in time to meet the next communications deadline each day.

In order to expedite the map-making and mosaic-making process, MIPL created a data ”pipeline.”

Organizing Millions of Bits of Information

Within 60 seconds of receiving data from Spirit and Opportunity, MIPL team members dribble and shoot thousands of bits of information into a ”pipeline.”

The pipeline processes all of the data from Mars into an automated assembly line that enables scientists, engineers, and the public to see and comprehend the martian world in all of its glory.

A pipeline pilot ensures that all the information flying into the system is heading to the right destination.

”The MIPL pipeline is like a gourmet catering service that preps, cooks, and serves packets of information from Mars into the right people’s hands on Earth almost instantaneously,” said Doug Alexander, MIPL team leader and development engineer.

The pipeline is the ”smarts” and ”big glue” that runs a lot of programs, Zamani explained. ”It says, ‘If I have a left pancam image, I need a right pancam image to make a 3D image.”’

Depending on rover power levels and the amount of time Spirit and Opportunity have to transmit data via the Odyssey orbiter relay or a direct-to-Earth antenna link, team members on Earth receive only some of the data during communications downlinks.

The pipeline looks for any missing pieces of information based on time codes and knowledge of when information should have been collected. It may take four or five days to get all the images down to create one panoramic mosaic.

In the meantime, beyond getting the images for a broad landscape, the MIPL team must get enough pieces of information every day to create the 3D maps, so ”the pilot watches the pipeline like a hawk,” Zamani said.

Straddling Science and Operations

”The quality and volume of data from the rover mission is high, and the trick is to organize the information quickly and have a pipeline that never breaks,” said Alexander.

The MIPL team developed methods to make sure that what the rover uplink team commanded the rover to send back to Earth did in fact come down the next day.

As the pipeline pilot does the checks and balances over tens of thousands of lines of data, other MIPL team members check the quality of the other 14 products. ”We straddle the fence between science and operations,” Alexander said.

Thrown into a small area called the ”bullpen” for 12-hour shifts, six MIPL team members are generating the fundamentals that make the rover function around the clock.

The skilled professionals are constantly creating 3D terrain meshes and ”gluing” together multiple pieces of data from the hazard-avoidance cameras (up-close images), navigation cameras (middle distance images), and panoramic cameras (far-away images) to give a different view of the martian landscape for multiple tactical purposes.

MIPL software products and team members even remove the fisheye view from camera images and create a human-eye view.

”The MIPL-engineered software has been optimized for processing rover camera data automatically, but the beautification of the mosaic products is a result of human intervention,” Alexander said.

The software developed at JPL can process camera data to quickly generate a mosaic, but detail-oriented, tireless team members must eliminate the doubling of the foreground rocks and match the brightness for one scene that may have been shot over multiple days.

Known as the Mosaic Czar, Bob Deen didn’t take one day off for the first 90 days of the rover mission.

”I was the general trouble-shooter at the beginning of the mission, and we all wanted to improve the software to generate products quicker for the rover drivers and press conferences, so it was too exciting to take a day off,” said Deen. ”The worst stress comes from the terrain meshes for the rover drivers because they can’t plan the traverse until we combine the terrain information from various cameras.”

Early in the mission, stress also flared when press conferences were happening every day. Scientists would often come to Deen and the MIPL team and request new mosaics or other special images at 5:00 a.m. for a press conference planning meeting at 6:00 a.m.

One night, after processing Mars images for 14 hours, Helen walked out of the office and noticed an odd iridescent glow in the distance. ”I thought, ‘gosh, that’s really strange — I wonder what that is,”’ said Helen. She later realized it was a band of the Sun rising on Earth . . . a sun she hadn’t seen for a few days working the martian day.

Dedicated Team

No one on the MIPL team complains about the long hours, or the fact that they are the ones hidden behind the curtain.

”I have always liked science fiction and computers,” Deen said. ”My dad had a business doing tax returns in the early ’70s on computers when computers were big mainframes in a room. I got an Apple II in high school and learned everything. I thought I wanted to work for Lucasfilm (of Star Wars fame), but when I found out in college that NASA hired people to run the computers for space science, I knew I wanted to do real science instead of science fiction. Now, after looking at all the rocks on Mars, I want to take a geology class,” Deen said.

”Our team has an age range of 21-year-old college students to people at the twilight of their retirement, and it’s interesting to see how everyone gets along with a common goal,” Alexander said.

The rover team is full of some of the brightest and most creative scientists and engineers who have to balance their various goals for their instruments.

”People work shoulder to shoulder with no envy, and they get over political work issues quickly in order to ensure the science gets accomplished,” Alexander explained.

Everybody makes sacrifices depending on the most efficient tasks of the day and how the team can use the various instruments to help solve the mysteries of history of water on Mars. ”It’s amazing how much human energy is behind a six-wheeled rover,” he said.

Rewriting Science Books

”When I was in grade school, I only saw blurry images of planets in our textbooks,” Mortensen said. ”Everybody is curious about the unknown, and one way to unveil that curiosity is to actually seek and give facts.”

With the image processing technology MIPL has developed for NASA, students can use the Internet as their textbook and see crystal-clear images from Mars within hours of the moment the picture was taken on Mars.

”For the missions I’ve worked on at NASA, including missions to Neptune and Jupiter, I have seen that technology actually goes into the public and ultimately benefits our entire society,” she said. ”Our current techniques for looking at data and analyzing the data are becoming faster and more sophisticated, and the commercial applications are infinite.”

Working with every planetary mission at JPL, MIPL is creating new views of our universe and changing the way we see the solar system and beyond.

”My advice to any student who is interested in space is to learn as much math and science as you can. You are always going to see something really cool if you just keep your eyes open.”

To learn about other star players on the MIPL team, please visit Zipcode Mars and type in the names of individuals in the team photo.