Big data is one of the big topics of the day with no shortage of statistics and predictions illustrating just how much data we will have to manage in the near future. Some of the organisations I speak to expect the amount of data they generate to double in just the next twelve months. For a storage guy, this sounds like great news, because not only is the volume of data increasing exponentially, but so is the velocity and variety of that data. And all of it will need to be stored, right? Not necessarily. What interests me is whether we can reach a point where the rate of data generation is continuing to increase, but the storage capacity trajectory is slowing down…and maybe even decreasing.

Probably the biggest challenge is the unstructured nature of modern information. Phone calls, blog posts, video feeds, and other types of user-generated content are examples of data whose

meaning is unintelligible for machine analysis, and requires some level of human intervention.

This creates challenges in the areas of efficiently storing, indexing, and deriving meaning from

data so you can support decisions, mitigate risk, and comply with regulations.

This type of unstructured information comprises more than 80% of the information in any enterprise, and it grows at more than three times the rate of structured sources. Gaining understanding from it is tricky, and traditional techniques like keywords and meta tagging have proved to be woefully inadequate.

Speed is of the essence

Then there is the question of speed. It’s no good assessing and analyzing data if you can’t do it in time for it to matter. Businesses have never been under so much pressure to make instantaneous decisions and react to competitive and customer shifts in real-time. This means that the right people need the right information at the right time. At HP we call this time-to-value, and the fact is that the window for it is shrinking at an alarming rate.

Once you have mastered the art of taking huge volumes of structured and unstructured data and extracting insight from it, you can decide on the value of that data to the business. Almost all of the customers I speak to tell me that they store all of their data, because they don’t know whether it is valuable or not. They don’t want to run the risk of losing something that they later need or will be penalized for disposing of. Searching and analyzing data lets you decide whether you need to keep it. And if you don’t need it, you don’t need to store it. This adds a new dimension to the meaning of storage efficiency—and lets you tame the beast of big data.