Big Data successes and limitations: What researchers and marketers need to know

This post identifies a few areas where big data is already making a difference and a few areas where it is going to make a difference soon. But the key focus of the post points out that the potential for Big Data is currently being overstated, and in terms of the short term, overstated massively.

The underpinning for the blog post is Nate Silver's book, The Signal and the Noise, probably the most comprehensive assassination of the creed of Big Data produced in the last 18 months - and a book I would recommend every marketer and market researcher to read.

Big Data successes

Big Data has had some high profile and impressive successes. Tesco, Target and Amazon, for example, have used Big Data to gain competitive advantages in retail. The Billion Prices Project from MIT has shown how monitoring the billions of posts on the Internet can be turned into hard data about prices for products around the world. Perhaps the strongest case for Big Data is the way that Google has used it to sell and target advertising.

In Nate Silver's book he points out how the massive use of computing and data have improved the quality of weather forecasting, especially over the last 25 years. However, Silver also points out that weather forecasting is limited in use and accuracy when applied to dates further away than one week. In this way, Silver indicates the power and the limitations of Big Data.

Yes, there are some Big Data successes. But have you noticed that the same few examples are talked about over and over? That is because they are still relatively rare, even though brands like Amazon, Tesco, and Google have been doing it for years.

Big Data successes are going to be limited over the next 5 years

There are a wide range of problems that Big Data will tackle over the next few years, but there are several factors that will slow down its development. These include:

Limited supply of data scientists and people capable of working with Big Data. A 2011 McKinsey & Company's Business Technology Office report said, "The United States alone faces a shortage of 140,000 to 190,000 people with deep analytical skills as well as 1.5 million managers and analysts to analyze big data and make decisions based on their findings."

The growth in noise is swamping the signal that businesses hope to find in the data. Nate Silver's take on the massive growth of data is that "[m]ost of it is just noise, and the noise is increasing faster than the signal."

There is a lack of models, making data less useful than it might be. People like Chris Anderson, editor-in-chief of Wired Magazine have claimed that Big Data will get rid of the need for theory and even the scientific method - but Silver shows this to be wrong and describes it as "categorically the wrong attitude." Until organisations realise that hypotheses, testing, and experimentation are essential, their Big Data gains will be limited.

The limits to big data in the longer term

There are many problems that can't be solved by simply collecting more data. Nate Silver illustrates this point with the problems in predicting earthquakes. Despite all the science, data, and models that have been thrown at predicting earthquakes, there has been no appreciable progress. It could be the data is insufficient, or that the models are incomplete, or that the system is too chaotic (in a mathematical sense) to make it capable of being modelled on the sort of information that could realistically be collected.

In terms of businesses, two problems are particularly relevant to the limitations of Big Data.

Endogeneity. Endogeneity refers to a feedback loop in a system. In social systems (such as economics, politics, and marketing), when we discover a 'law', i.e. something that is true, it starts to become less true. Economists have discovered all sorts of "rules" about the economy, explaining inflation, recession etc. But when policy makers try to use these rules, they often don't work. People and markets know about the rules, they factor them into their responses, changing the rules. Einstein's quote that "No problem can be solved from the same level of consciousness that created it" describes this sort of problem. We can model and predict ants or algae far better than we can model and predict markets.

The ratio of cases to observations. Nate Silver illustrates this problem when he explains one of the reasons why the forecasting of economic recessions is so bad. The US has had 11 recessions since World War II, but there are millions of potential social and economic indicators (the US Government tracks 45,000 economic indicators, private organisations as many as four million). With this many indicators, most of the patterns are going to be spurious. We'd be better off (from a maths perspective) if we had 11 indicators and hundreds of recessions.

When is Big Data relevant?

Big data is most useful when there is a clear question and where models of behaviour can be created and tested. In the US, Target asked the question "Can we figure out if a customer is pregnant?" and produced a model that was a) quite predictive, and b) got them into the pages of New York Times Magazine.

At the 2013 MRS Conference in London, Lucien Bowater Director of Strategy and Insight at BskyB said that when he knows where to dig, Big Data is great. So, he uses market research to tell him where to dig, and his data people to do the digging.

In the world or marketing and market research, the impact of Big Data is going to be more profound in connecting the dots, rather than by creating whole new pictures.

Using research with Big Data

The role for market research in the future is likely to be to provide a prism for Big Data, showing the data scientists where to dig, helping create the hypotheses to test, and adding a "why" to Big Data's "what." In Nate Silver's book, he shows how people with insight working together with Big Data perform better than just Big Data or just insight. He shows that baseball scouts working with systems like his PECOTA system beat just Scouts and just Big Data.

At Vision Critical, we are working to integrate insight communities into the Big Data picture. In an insight community, we can follow people over time to help find patterns, build and test hypotheses, and have discussions with members to explore the whys. This work can be interfaced with Big Data to quantify, validate, and evaluate the insights derived by combining the insight community with the data sources.

What marketers and market researchers need to do next?

Marketers and market researchers need to get to grips with Big Data. There are going to be some successes, and we want to be involved in those. Also, ignoring something that so many "experts" are saying is going to be huge makes you look foolish (even if you are being smart).

My forecast is that over the next five years, most Big Data projects (in the areas of sales, marketing, and brand growth) will not make a significant, positive contribution to the teams, organisations, and companies commissioning them.

The projects that will work are going to, typically:

Ask a very clear, definite question

Include a testable model, based on hypotheses

Use market research to scope, define, and interpret the findings and processes

Is big data a waste of money? Not necessarily. But for Big Data projects to be meaningful, researchers and marketers need a strategic approach that involves listening to what customers have to say.