Founded in 1993 by brothers Tom and David Gardner, The Motley Fool helps millions of people attain financial freedom through our website, podcasts, books, newspaper column, radio show, and premium investing services.

Is Big Data Really a Big Deal?

Enormous data sets are everywhere. The real trick is to squeeze valuable information out of the raw data.

Modern society is awash in massive data sets. I'm not just talking about the Internet, although that's certainly the most obvious and diverse example you'll encounter all day. The data sources are unending:

Every phone call and text message becomes a data point in your carrier's extensive logs. Multiply this by literally billions of global landlines and cell phones, and the Internet starts to look small by comparison.

I once drew a map showing how to get from Miami to Jacksonville, Fla.: two points connected by a straight line representing I-95. Though that was a memorable achievement, reality is complex and maps can get incredibly large and detailed.

Weather forecasts have moved far beyond reading the Farmer's Almanac and looking at the sun. Weather stations are constantly collecting heaps of data and passing it on to supercomputers for processing. And we still can't see something like Hurricane Katrina coming until it's too late to do much about it. Those spaghetti models sure look pretty, though.

Wal-Mart is said to own the largest database anywhere, generated by trillions of transactions at history's biggest retailer. Smaller retailers naturally have smaller data sets to analyze, but sometimes they're richer: Amazon.com(Nasdaq: AMZN) goes beyond anonymized purchasing habits by knowing what your email address is, where you live, and how you click around its sites.

These are just a few examples from a much longer list. The list of Big Data generators is a pretty big data set in itself.

Add it all up, and market-research firm IDC says you got 180 exabytes of information in 2006, or 180 billion gigabytes. That tally rose by 56% in 2007 and is expected to multiply by 10 between 2006 and 2011.

Big problem? Big business!If you find it hard to wrap your head around a number that large, imagine the desperation of business managers needing to mine this wealth of data to find nuggets of actionable information. Or you can visualize IT professionals breaking out in cold sweats over the scope of that task, or in bouts of evil chuckles as they realize the equally enormous business opportunity of it all.

You need to store it, which is good news for leading storage vendors EMC and NetApp(Nasdaq: NTAP). They go a bit further than just slapping together enormous disk arrays for these uses and also make software that eliminates duplicated data points. Data deduplication, as this is known, can be a major selling point in certain Big Data environments and has become the focus of at least two bidding-war acquisitions in recent years.

Beyond storing it all, you also need to move the data to where it's needed. That can mean streaming video to a consumer or collecting data in a centralized data warehouse. Either way, there's a network involved. Yep, networking experts from Cisco Systems(Nasdaq: CSCO) to Brocade Communications Systems are licking their lips, too, and are happy to optimize their systems for these cases. Cisco expects machine-to-machine traffic to increase at a compounded growth rate of 258% through 2015. Oh, yeah, it's a big deal.

And then you need to analyze it, which is the real black magic of it all.

The real gusherData mining is the art and science of looking at mostly meaningless data to divine actionable patterns, and business intelligence is data mining applied to, well, business. This is where the real opportunity lies.

Wherever there's a database, you'll find Oracle(Nasdaq: ORCL) and IBM(NYSE: IBM) looking for business. Both sell dedicated data-mining and business-intelligence products, and they're pouring more research dollars into this sector than anybody else. But they're already humongous companies for which it's hard to move the needle very far, even with an opportunity of this magnitude.

For us investors, it's better to find smaller specialists in the field. It's easier to double a billion in sales than a hundred billion, after all, and the stocks are primed to follow the same pattern. So pinpointing thought leaders on a much smaller scale will maximize your returns.

Or perhaps the better opportunity lies in data-mining generalist Teradata(NYSE: TDC). When Wal-Mart needed a hand with the transaction data we mentioned earlier, mid-cap Teradata was the only company qualified to do the job. Top-notch data miners are few and far between, and Teradata is about as good as it gets.

Tibco and salesforce.com have been identified as shorting opportunities by one of our own Foolish newsletter services, even while another newsletter recommends buying salesforce.com. I disagree with the shorting theses because I see heaps of hidden value left to unlock in both stocks.