Is big data science fiction?

Today, the promise of big data is less fantasy and more reality

By Evan Stubbs, Chief Analytics Officer, SAS Australia / New Zealand

Our world is a fascinating one; we’re at an inflection point, one defined by big data and business analytics. What was once science fiction is becoming reality. Let’s be frank though—that sounds pretty hack­neyed. After all, hasn’t everything been science fiction once?

This is true. It’s also true, however, that science fiction is a deep well to draw from. A well where some ideas are so fantastical that it seems impossible that they’ll ever become reality. Asimov, a science fic­tion writer, for example, wrote speculatively of “psychohistory” in his Foundation series. A form of mathematical sociology, scientists would use massive amounts of behavioral information to predict the future.

Through doing so, they were able to foresee the rise and fall of empires thousands of years in advance.

As with all good stories, power always comes with constraints. Accurate predictions were only possible given two conditions. First, the population whose behaviors were to be modeled needed to be suf­ficiently large—too small, and the predictions would become error-prone. Second, the population being modeled could not know it was being modeled. After all, people might change what they were doing if they knew they were being watched.

It seems fantastical, doesn’t it? Still, this is fundamentally the promise of big data.

Big data offers unparalleled insights and predictive abilities, but only to those who know how to leverage it.

We know more about the world than ever before. Many of those being watched are still unaware of how much things have changed. Between national intelligence, security leaks, and the potential of metadata, most of us are only just realizing how much infor­mation is out there. And, by analyzing that data, we have the power to predict the future in ways that people still can’t believe. Amazon, for example, took out a patent in late 2013 on a process to ship your goods before you’ve ordered them.

Big data offers unparalleled insights and predictive abilities, but only to those who know how to leverage it. For most, getting value from big data is a challenge. However, the reflec­tion of every challenge is opportunity.

Things have changed. And, it’s a rare leader who isn’t aware he or she needs a plan to realize this opportunity. However, there’s a twist. It’s not just a good idea. It’s not something that’s going to happen. It’s happening now.

Catalyzed by books such as Thinking, Fast and Slow and Nudge, behavioral economics is already blending data with heuristics and psychology to create new models to describe and influence consumer behavior. Recognizing the power of a scientific approach to analyzing information, the U.K. government established a dedicated Behavioral Insights team to take advantage of these ideas. Formed in 2010 and nicknamed the “nudge unit,” their goal was to blend quantitative and qualitative techniques to improve policy design and delivery.

Paul Krugman, winner of the Nobel Memorial Prize for Economic Sciences, credits Asimov’s vision of a mathematical sociology as inspir­ing him to enter economics. This vision of a future shaped by our abil­ity to analyze information is becoming real. And, it’s changing the face of medicine, policy, and business.

Thanks to constantly increasing ana­lytical horsepower and falling storage costs, the cost of sequencing the genome has dropped from US$100 million in 2001 to just over US$8,000 in 2013. More than just being cheaper, every decline in sequencing costs puts us that much closer to truly personalized medicine.

Even the social web is sparking innovation. Facebook’s acquisition of Oculus, Instagram, and Whatsapp wasn’t just an attempt to diver­sify. It was a deliberate attempt to stay engaged across all channels all the time. With over a billion people now on Facebook, it’s amazing what one can find by scanning personal interactions.

Organizations like the United Nations (UN) are tracking disease and unemployment in real time through the large-scale analysis of social media. The Advanced Computing Center at the University of Vermont is using tens of mil­lions of geolocated tweets in its Hedonometer project to map happi­ness levels in cities across the United States.

The future is closer than it’s ever been. Taking the leap to Asimov’s psychohistory isn’t as far-fetched as it once might have seemed.

Evan Stubbs is the Chief Analytics Officer for SAS Australia and New Zealand. He authored three books about business analytics, sits on the board of Institute of Analytical Professionals of Australia, and is a guest lecturer at Macquarie University and the University of Sydney.

He is a recognized expert in innovation and leads the Advisory business within SAS Australia, a group focused on transforming organisations into analytical competitors. His practical and experience-based talks on creating value through the use of business analytics are in high demand and feature regularly as keynote presentations.