Big Data? Big Whoop.

Big Data? Big Whoop!

Over the past couple of days I’ve been seeing a number of posts and tweets about Biiiiig Dataaaaaaa (ring announcer voice in my head)! What is “Big Data”? Check out the definition in this executive summary; or as I and others like to say, “It’s as big as a piece of string is long”. I certainly understand the idea behind “Big Data”, but do we really need a new term for something that, let’s face it, isn’t new at all?

In a comment to this post I used the phrase “E2.0 meets BI”. To be more accurate I should have said “E2.0 fuels BI”. This whole “Big Data” thing is nothing more than reporting and analytics, but with more data than we had before. Those of us who have a stake in the BI domain have often wished for more raw data on which to base our decisions. Now that we have it, and are getting more of it every second, we’re freaking out and giving one or more major vendors in the space an opportunity to define something new. Two things, and only two things, have really changed:

The available amount of raw data is way beyond what it was only a short time ago;

The Cloud and SaaS jeopardize access to some of the raw data.

If you’ve got the resources (i.e.: $’s) dealing with #1 one is a matter of scaling. Dealing with #2 is tougher, especially if any of your data sources are not entirely under your control (Cloud, SaaS). The challenge, however, is not insurmountable:

Rationalize your requirements and identify what is absolutely critical to your business (i.e.: leave the “it’s just cool” stuff out or defer for later);

If you rely on hosted data sources negotiate appropriate access and up-time agreements;

Find out if your hosted providers can provide some of the ETL for you;

If you have retention / disposition policies on your data sources, enforce them; if not, define and enforce them.

The funny thing is, when I made my living in BI projects, 5 of the 6 points noted above where standard things we did. Maybe nothing really has changed all that much, other than my segue into RIM. Oh well.