What will big data disrupt in five years?

By Frank Konkel

Aug 13, 2013

Examples of big data being harnessed at the federal level abound.

Every piece of mail sent in America is photographed, monitored and analyzed by supercomputers; the Centers for Disease Control and Prevention mines social media data to track and predict disease outbreaks in real time, and perhaps the most complex big data operation on Earth, maintained by the National Security Agency, analyzes vast amounts of Internet data and phone records for foreign signals intelligence.

Yet we are squarely in the early stages of the big data bang, and developments today offer a glimpse of how ingrained in federal operations big data might be in the next five to 10 years, according to several industry and government experts speaking during Meritalk's Big Data Forecast webinar Aug. 13.

Dr. Sasi Pillay, chief technology officer in NASA's Office of the CIO, said the space agency is beginning to switch from a reactive model to a predictive one regarding big data—or in an analogy he offered drawing from science, moving from "earthquake reactionary" to "tornado watch." Under the new model, algorithms, modeling and simulations – complete with advanced visualization tools – are helping the agency reduce testing of aeronautics equipment, potentially saving billions of dollars.

New aeronautics equipment comes with modern instrumentation that collects data multiple times per second, which ultimately helps NASA build better machines. Big data innovations are seeping into NASA's IT security division as well. The agency is "feeding in what is happening on social media" and applying that data stream to others, enriching information collected from isolated data points.

In addition, Pillay said, NASA is looking to big data to improve efficiency of airline traffic, comparing metrics such as airline routes, the density of flight patterns and fuel consumption. In some instances, NASA also makes use of large three-dimensional data visualizations, creating an "immersive" environment scientists "can walk around in."

Other fields are going to get a boost from big data, too.

Rich Campbell, chief technologist at EMC Federal, said big data will dramatically change how cyber-security is implemented and monitored as well as how digital forensics are carried out. In the broad scheme of federal agencies, Campbell said, efficiencies and processes will be streamlined through vast number crunching and smart algorithms.

Big data allows insights at the granular level, Campbell said, explaining how disparate data sets from a call center – how many customers called, how many claims were processed and how fast, and net results from calls – could be compared and used to make decisions.

Campbell also said the importance of metadata – commonly referred to as the "data about data" – will continue to grow. Appropriately tagged data could be shared across agencies, with organizations outside government or with the public itself. He called metadata a "big future trend."

"It's not just how we put our hands around it, but how we tag it appropriately" so it can be used, Campbell said.

The leading innovators of big data continue to be in the intelligence community, and there is little reason to doubt that advancements in cyber warfare will come through big data.

Sam Lee, data center practice director at Force 3, said cyber warfare efforts fit well in big data's space. The ability to store, sift and analyze huge amounts of Internet traffic at the point of attack is decidedly a big data effort, and could play an especially important role in the near future given the complexity of today's cyberattacks.

Big data is only going to get bigger in the future, too.

Pillay provided an outline of the Square Kilometer Array (SKA) project, which will create the world's largest telescope. When the telescope is operational – approximately a decade from now – it will create 1 million terabytes (one exabyte) of data per day, or double the amount of total worldwide Internet traffic each day. Managing that data won't be NASA's problem – the project doesn't involve the United States – but it will be as large a big data challenge as there's ever been.

Reader comments

Thu, Aug 15, 2013
Joe

What the government can do is disrupt private lives and political movements, as it is doing with the IRS scandal. If men were angels, this wouldn't be an issue. There are many things that would be useful, but I have no trust in either government that is here to help me. Will they track trends in how they are manipluating people with the latest thing about using phrases and words to "help us make better choice"? I think NOT.

Wed, Aug 14, 2013
OccupyIT

The real disruption for most agencies will be the amount of money wasted on 'big data' because its cool and everyone thinks they want to do it... Even if they don't have, use, or need 'big data' in their program. Have Hammer.... Find Nail.... Forget Mission...

Wed, Aug 14, 2013

Metadata for example is a day trader looking at all stock quotes for a day and matching that with prior purchases after quotes and resources to purchase stocks. Once the day trader gets a good relationship in the metadata analysis, they can put in their own stock purchase or short. Or maybe it will be the Trreasury Department and they want to know where the market is going on 10 year notes. They see a blip and boom, they either print more or they buy. It's just the government doing it; it is not market manipulation is it?

Please post your comments here. Comments are moderated, so they may not appear immediately
after submitting. We will not post comments that we consider abusive or off-topic.