Featured Research

from universities, journals, and other organizations

Managing the deluge of 'Big Data' from space

Date:

October 21, 2013

Source:

NASA/Jet Propulsion Laboratory

Summary:

For NASA and its dozens of missions, data pour in every day like rushing rivers. Spacecraft monitor everything from our home planet to faraway galaxies, beaming back images and information to Earth. All those digital records need to be stored, indexed and processed so that spacecraft engineers, scientists and people across the globe can use the data to understand Earth and the universe beyond.

Share This

For NASA and its dozens of missions, data pour in every day like rushing rivers. Spacecraft monitor everything from our home planet to faraway galaxies, beaming back images and information to Earth. All those digital records need to be stored, indexed and processed so that spacecraft engineers, scientists and people across the globe can use the data to understand Earth and the universe beyond.

Related Articles

At NASA's Jet Propulsion Laboratory in Pasadena, Calif., mission planners and software engineers are coming up with new strategies for managing the ever-increasing flow of such large and complex data streams, referred to in the information technology community as "big data."

How big is big data? For NASA missions, hundreds of terabytes are gathered every hour. Just one terabyte is equivalent to the information printed on 50,000 trees worth of paper.

"Scientists use big data for everything from predicting weather on Earth to monitoring ice caps on Mars to searching for distant galaxies," said Eric De Jong of JPL, principal investigator for NASA's Solar System Visualization project, which converts NASA mission science into visualization products that researchers can use. "We are the keepers of the data, and the users are the astronomers and scientists who need images, mosaics, maps and movies to find patterns and verify theories."

Building Castles of Data

De Jong explains that there are three aspects to wrangling data from space missions: storage, processing and access. The first task, to store or archive the data, is naturally more challenging for larger volumes of data. The Square Kilometer Array (SKA), a planned array of thousands of telescopes in South Africa and Australia, illustrates this problem. Led by the SKA Organization based in England and scheduled to begin construction in 2016, the array will scan the skies for radio waves coming from the earliest galaxies known.

JPL is involved with archiving the array's torrents of images: 700 terabytes of data are expected to rush in every day. That's equivalent to all the data flowing on the Internet every two days. Rather than build more hardware, engineers are busy developing creative software tools to better store the information, such as "cloud computing" techniques and automated programs for extracting data.

"We don't need to reinvent the wheel," said Chris Mattmann, a principal investigator for JPL's big-data initiative. "We can modify open-source computer codes to create faster, cheaper solutions." Software that is shared and free for all to build upon is called open source or open code. JPL has been increasingly bringing open-source software into its fold, creating improved data processing tools for space missions. The JPL tools then go back out into the world for others to use for different applications.

"It's a win-win solution for everybody," said Mattmann.

In Living Color

Archiving isn't the only challenge in working with big data. De Jong and his team develop new ways to visualize the information. Each image from one of the cameras on NASA's Mars Reconnaissance Orbiter, for example, contains 120 megapixels. His team creates movies from data sets like these, in addition to computer graphics and animations that enable scientists and the public to get up close with the Red Planet.

"Data are not just getting bigger but more complex," said De Jong. "We are constantly working on ways to automate the process of creating visualization products, so that scientists and engineers can easily use the data."

Data Served Up to Go

Another big job in the field of big data is making it easy for users to grab what they need from the data archives.

"If you have a giant bookcase of books, you still have to know how to find the book you're looking for," said Steve Groom, manager of NASA's Infrared Processing and Analysis Center at the California Institute of Technology, Pasadena. The center archives data for public use from a number of NASA astronomy missions, including the Spitzer Space Telescope, the Wide-field Infrared Survey Explorer (WISE) and the U.S. portion of the European Space Agency's Planck mission.

Sometimes users want to access all the data at once to look for global patterns, a benefit of big data archives. "Astronomers can also browse all the 'books' in our library simultaneously, something that can't be done on their own computers," said Groom.

"No human can sort through that much data," said Andrea Donnellan of JPL, who is charged with a similarly mountainous task for the NASA-funded QuakeSim project, which brings together massive data sets -- space- and Earth-based -- to study earthquake processes.

QuakeSim's images and plots allow researchers to understand how earthquakes occur and develop long-term preventative strategies. The data sets include GPS data for hundreds of locations in California, where thousands of measurements are taken, resulting in millions of data points. Donnellan and her team develop software tools to help users sift through the flood of data.

Ultimately, the tide of big data will continue to swell, and NASA will develop new strategies to manage the flow. As new tools evolve, so will our ability to make sense of our universe and the world.

More From ScienceDaily

More Computers & Math News

Featured Research

Mar. 3, 2015 — By examining the forces that the segments of mosquito legs generate against a water surface, researchers have unraveled the mechanical logic that allows the mosquitoes to walk on water, which may ... full story

Mar. 3, 2015 — Major cities in the UK are falling behind their international counterparts in terms of their use of smart technologies, according to a new study. The research has found that smart cities in the UK, ... full story

Mar. 3, 2015 — To simulate chimp behavior, scientists created a computer model based on equations normally used to describe the movement of atoms and molecules in a confined space. An interdisciplinary research ... full story

Mar. 3, 2015 — Magnetic vortex structures, so-called skyrmions, could in future store and process information very efficiently. They could also be the basis for high-frequency components. For the first time, a team ... full story

Mar. 2, 2015 — The odds of picking a perfect bracket for the NCAA men's basketball March Madness championship tournament are a staggering less than one in 9.2 quintillion (that's 9,223,372,036,854,775,808), ... full story

Mar. 2, 2015 — Scientists report that they could observe experimentally the current flow along channels at the crystal surfaces of topological insulators. The channels are less than one nanometer wide and extend ... full story

Mar. 2, 2015 — Organic light emitting diodes (OLEDs), which are made from carbon-containing materials, have the potential to revolutionize future display technologies, making low-power displays so thin they'll wrap ... full story

Mar. 2, 2015 — What if one day, your computer, TV or smart phone could process data with light waves instead of an electrical current, making those devices faster, cheaper and more sustainable through less heat and ... full story

Mar. 2, 2015 — 3-D printing could become a powerful tool in customizing interventional radiology treatments to individual patient needs, with clinicians having the ability to construct devices to a specific size ... full story

Featured Videos

Forensic Holodeck Creates 3D Crime Scenes

Reuters - Innovations Video Online (Mar. 3, 2015) — A holodeck is no longer the preserve of TV sci-fi classic Star Trek, thanks to researchers from the Institute of Forensic Medicine Zurich, who have created what they say is the first system in the world to visualise the 3D data of forensic scans. Jim Drury saw it in operation.
Video provided by Reuters

Related Stories

Jan. 26, 2015 — Many satellites that monitor the Earth orbit relatively close to the planet, while some satellites that monitor the sun orbit our star. DSCOVR will keep an eye on both, with a focus on the sun. To ... full story

Oct. 8, 2012 — Technicians and engineers are completing final system checks and spacecraft inspections on the first of NASA's third-generation Tracking and Data Relay Satellites (TDRS). Boeing Space Systems ... full story

Aug. 3, 2012 — Screaming through the thin Martian atmosphere at hypersonic speed, the Mars Science Laboratory is about to experience heat and pressure up to three times more intense than when a space shuttle ... full story

June 9, 2011 — Newly-released portraits show the International Space Station together with the space shuttle, the vehicle that helped build the complex during the last decade. The pictures are the first taken of a ... full story

ScienceDaily features breaking news and videos about the latest discoveries in health, technology, the environment, and more -- from major news services and leading universities, scientific journals, and research organizations.