What To Do When Your Big Data Isn't Big Enough

Gartner predicts that the cumulative Big Data spend for the period within 2011 and 2016 will reach a whopping $232 billion.

Storage of Big Data--and particularly unstructured data such as video, PPT presentations, and Word documents that do not fit into a database--has been the subject of considerable discussion. While proper, secure storage was a natural evolution in the Big Data debate lifecycle, it is now merely a precondition to much higher orders of analysis and business intelligence. In this respect, the Big Data remains too small in many minds.

The proverbial cowboy--all hat, no cattle--has been flipped on its head in the data world. Enterprises have petabytes or exabytes of data, but often don’t know how to use it or have too few (or no) channels by which to translate those assets into actionable business intelligence. This is certainly the case with Big Data.

So when is Big Data not big enough?

The answer is simpler than one thinks, and often has nothing to do with data volume. Your Big Data isn’t “big enough” when it doesn’t address your business challenges as a result of a lack of either (i) useful information therein or (ii) the proper tools to look inside it and infer useful intelligence. The second usually proves to be the road bump. According to renowned
AT&T Bell Telephone Laboratories statistician Bill Tukey (1915-2000), data may not contain the answer and “an aching desire for an answer will not ensure that a reasonable one can be extracted from an existing body of data.” While Tukey is correct in an important respect--one should not overvalue data per se--he was never exposed to Data Science and business intelligence as they exist today. If he were so exposed, it’s not a stretch to posit that Tukey would warn us of a much greater danger -- having access to the proper data sets (both proprietary and open) but still failing to effect inefficiencies in an enterprise’s business processes.

While deriving actionable business intelligence from data is no easy feat even with the right tools, the use of open information platforms (often known as “crowdsourcing”) exposes companies to hundreds of thousands of global experts in data science, application development, and design. Appirio, for example, boasts a growing community of 660,000 experts. Working in an open innovation setting, an enterprise can leverage Data Science in order to create complex algorithms that are run against its data sets to find nuggets of actionable intelligence to drive business. This is the centerpiece of business intelligence, and its effects can be felt in every aspect of one’s company as its use becomes standard practice.

According to Gartner, through 2017, 90% of information assets will remain “siloed and unleverageable” across multiple business processes. This bears repeating: 90%. No enterprise can afford to let its data get stale by omission. Rather, companies must leverage one of their most critical assets (data) as a core part of their analytics efforts to outcompete the field.

The Big Data “dilemma”--too big or too small--depends on context and historical time frame. The “too big” problem (storage) has been widely debated with myriad solutions now in place. While too many enterprises still fail to follow what might be regarded as standard advice--the use of secure private clouds such as those available on Amazon’s Web Services, which has a 99.9999% uptime and world-class security protocols--that debate is magnitudes smaller than even two years ago.