Size doesn’t matter -- it’s the complexity of any data set that makes it big, not how many petabytes of storage it takes. The richer and more multifaceted your data, the more value you can extract. But you have several challenges to conquer before you can wring out every drop.

Shutterstock

In this, the second in our four-part series on big data-powered AI, we’re taking a deep dive into the first challenge: complexity. First, we’ll take a look at the threefold complexity that’s preventing companies from fully leveraging their data. Then we’ll define a powerful framework that will enable you to extract maximum value from that complexity.

The Complexity Challenge

By now, complexity should be part of the definition of big data. That complexity is matched only by the complexity of the technology needed to handle it, and the algorithms required to analyze it.

The famous components of big data’s intrinsic complexity (volume, velocity, variety) are not themselves complex. It’s their interaction -- the way they act upon and transform each other -- that engenders the intrinsic complexity. The volume, variety and velocity of big data conceal valuable information -- assuming the machines and algorithms can keep up.

The technology needed to process and manage the big data is equally complex (hardwired complexity). New technologies are springing up at every moment to process the explosion of data (e.g., Hadoop, Presto, Spark, Storm), and entire companies have come into being just to manage it (e.g., Hortonworks, Splunk, MapR, Cloudera).

But even after you’ve conquered the hardwired complexity of the technology, your data is useless unless you’ve got the algorithmic complexity to dig out the insights buried within it. Lacking the sophisticated data science, intricate ML algorithms, intuitive AI and human talent needed to extract patterns and predictions from your data, it has no business value. But by leveraging the three M's (methodology, machine and manpower) that we propose, you can harness complexity to solve specific business problems.