Big data noise has reached the point where most are reaching for the ear plugs. And with any good hype bubble, the naysayers are now grabbing attention with contrarian positions. For example, The New York Times expressed doubt about the economic viability of big data in "Is Big Data an Economic Big Dud?" This post grabbed a lot of attention, but, like many others I read, it fundamentally misses the point of what big data is all about and why it's important. The article compares the productivity boom associated with the first wave of the Internet to the lack of growth experienced since the inception of "big data"; it implies that big data’s expected economic impact may not happen. Furthermore, the article implies that big data is something that firms will do or implement. Thinking about big data this way or differentiating between data sets as big, medium, or small is dangerous. It leads to chasing rabbits down holes.

Here is what I mean. We all know that big data is getting bigger — I just read another NYT post that projects 40 trillion gigabytes of global data production by 2020. Awesome, but so what? Most firms I talk to have only a vague notion of what could be done with all this data, yet they are ramping up big investments in big data hoping to capture insight from new social and mobile data sets. After six months to a year of dinking around and being disappointed with their "build big data and they will come" strategy, the business will stop funding things and IT will get another black eye. Nobody needs that.

This brings me to the myth about big data that I debunk the most — that it's all about social and mobile data gone out of control. In Forrester's Forrsights Strategy Spotlight: Business Intelligence And Big Data, Q4 2012 survey of 634 BI users and planners, we found that social and mobile data is actually a pretty low priority:

While big data technologies like Hadoop can definitely deal with over-hyped new data like mobile and social, the broad demand is simply not there yet. Instead we find leaders recognize that the big deal about big data is the potential for getting more value more quickly from more data, at a lower cost and with greater agility; and it is a whole range of technologies and new techniques that make more possible.

BUT . . . just because you can doesn't mean you should. Tomorrow's winners understand this, and they are making selective investments aimed at specific opportunities with tangible benefits where big data offers a more economical solution to meet a need. For example, we compared firms that were growing 15% or more year-on-year to their peers in our Forrsights Strategy Spotlight: Business Intelligence And Big Data, Q4 2012, and found that:

High-growth firms are more interested in what's possible tomorrow than what's painful today. For example, high-growth firms were 49% more likely to be concerned about data velocity and 23% more likely to be concerned about data access limitations of older-generation technology. The pack was concerned about data cost and not being able to react fast enough.

High-growth firms are investing more aggressively than their peers. For example, high-growth firms were 53% more likely to be expanding or upgrading their big data investments this year than the rest of the pack.

High-growth firms are much more likely to be investing in next-generation technologies. For example, high-growth firms were only 6% more likely to be investing in relational database technology than their peers. But these same firms were 54% more likely to be investing in Hadoop and a whopping 156% more likely to be investing in other NoSQL databases.

Furthermore, I've found, in my direct interactions with firms, that leaders know the real value of big data is lower cost and greater agility. When I interviewed 11 firms with production-class big data implementations, all of them told me the same thing — they got into big data when they couldn't figure out how to accomplish what their business wanted in an affordable way with their existing technology. Further, none of these firms started with external social, mobile, or other exotic forms of data. They began with data they had and understood but could not afford to capture and analyze at scale.

The point is that high-performance firms seem to be on to something in big data that many miss. It's not about the hype, it's about the best solution to meet needs most effectively. They are opportunistically finding ways to exploit new technology and analytic techniques to get past limitations, lower cost, and create greater agility; and they are focusing on the data they have while looking opportunistically toward the potential for adding newer internal and external data sets into the mix.

This is the big deal about big data, but the story is only getting started.

We wrote about big data hubs in our report, Deliver On Big Data Potential With A Hub-And-Spoke Architecture, and indeed we think most firms will shift from a data-warehouse-centric approach to one that favors low-cost data hubs. But the impacts of the new data economy, open data, the API economy, and cloud analytics are yet to be felt. In the meantime, big data will continue to cannibalize older-generation technology approaches, and vendors will keep at the "me too" marketing message to sustain their revenue stream. Eventually, once firms stop trying to fit square pegs into round holes, we will see big data deliver economic benefit as firms become much more responsive to the opportunities that digital information opens to them. We won't be able to say that big data was the only reason, as there are other factors like cloud that are creating tomorrow's ferocious, and nimble, competitors.

While all this works out, the best advice I can give firms is to dump their isolated, IT-led "big data strategy" aspiration, and embrace the potential that the big data concept implies. Rather than separate big data or even data strategies, I encourage firms to ensure that a plan for data is a part of their business strategy, and this plan must account for the new reality — we have the tools and techniques to enable agility while we affordably operate a larger scale. At the end of the day, that's what big data is all about.

Good morning Brian. Thanks for the response. I did see that but wanted to make sure that flexibility - or agility - wasn't seen as soley a scale based thing. People get fixated on the "big" and many vendors without actual experience tend to talk Petabytes rather than actual deployment patterns.

In any case great post and thanks for helping to get what reality actually looks like into circulation.

Brian, thanks for calling out a story that's too seldom told, that big data can have a big impact in agility and operational excellence. In my experience with advising enterprises in adopting new technology, I'll suggest that use cases vary. The journos just tell the big story of the moment as implied here.

My current advice for CIOs and CMOs considering big data is, "Don't worry about 'what kind of data' or technology or bigness ;^). Start with what are the stakeholders you want to influence with 'better decision making/experience'?" If those stakeholders are all over social media, that data will be critical. If your reverse logistics are causing angst, focus on back house data. This information would qualify the results of the survey you cite. The other thing is, very few respondents probably understand some of the emerging data types or how to use them; most big firm analytics departments are still staffed with people who prefer structured data.

Lastly, I hypothesize that, since the general desire is adding "unique value," most orgs will accomplish that by creating cocktails with their proprietary (internal, structured) data and stakeholder-relevant external data, and they'll use it to support stakeholder outcomes of using the company's products & services.

Think this was spot on in right direction, so wanted to add brief comment. Certainly agree that it is a very good thing for most orgs if they do not yet have a 'big data' strategy--could go on for hours why, but reduced is combination of factors surrounding physics of data convergence, & what that will mean in the next five years relative to emerging technology.

I prefer the word adaptive to agile (founded what may have been first agile lab & incubator post web commercialization).

Reason is subtle, but important. I've observed many companies of all sizes & types practice agility with fair amount of athletic ability, but jump in the wrong direction--sometimes over a cliff or right into the fire. This seems to be occurring frequently today with big data tactics & projects, hype surrounding speed, apparently driven in part by short-term pressure & misaligned performance incentives.

While obvious that isn't what most mean by agility, it did have impact on my thinking, including R&D and architecture. The meaning I intend with adaptive is relative to need, as changing conditions & good quality data/information warrant, on continuous basis for each entity. Some data requires real-time action, some requires long-term strategic planning, but whatever that data suggests it better be of good quality or horrendous mistakes can be made; as we all see regularly.

We just completed this report with title that is a good example of my own interpretation, the first of several in the works for leaders in each industry.

The data market is booming with an ever increasing valuation. In the virtual data ocean, Big Data and small data are complementary strategies at one level and choices of scale/scope at a different level. While Big Data holds distant promises, leveraging “small data” can provide great benefits to small-to-medium business (SMB) as well as large corporations. Simply stated, the cost-performance barrier leaves many lucrative markets inadequately served by Big Data approaches. Carefully chosen data solutions and models should be more accurate, objective, and ultimately lead to improved ROI especially for more near-term time horizons and/or companies with limitations in scale/scope.