Top 5 Big Data Challenges

In the recent past, we were constrained by many limitations around data. Now, we are only limited by our imagination. By using more data and more types of data, we can fundamentally transform our organizations, and our world. These transformations bring great opportunity, but also come with challenges. To that end, here is my take on the top 5 big data challenges organizations face today:

The biggest challenge by far that we see with Big Data is that it is difficult to find and retain the resources skills to staff Big Data projects. The fact that “Its expertise is scarce and expensive” is the #1 concern about using Big Data according to an Information Week survey of 541 business technology professionals (1). And according to Gartner by 2015, only a third of the 4.4 million big data related jobs will be filled (5)

2) It takes too long to deploy Big Data projects from ‘proof-of-concept’ to production

At Hadoop Summit in June 2014, one of the largest Big Data conferences in the world, Gartner stated in their keynote that only about 30% of Hadoop implementations are in production (4). This observation highlights the second challenge which is that it takes too long to deploy Big Data projects from the ‘proof-of-concept’ phase into production.

3) Big data technologies are evolving too quickly to adapt

With the related market projected to grow from $28.5 billion in 2014 to $50.1 billion in 2015 according to Wikibon (6), Big Data technologies are emerging and evolving extremely fast. This in turn becomes a barrier to innovation since these technologies evolve much too quickly for most organizations to adopt before the next big thing comes along.

4) Big Data projects fail to deliver the expected value

Too many Big Data projects start off as science experiments and fail to deliver the expected value primarily because of inaccurate scope. They underestimate what it takes to integrate, operationalize, and deliver actionable information at production scale. According to an InfoChimp survey of 300 IT professionals “55% of big data projects don’t get completed and many others fall short of their objectives” (3)

Uncertainty is inherent to Big Data when dealing with a wide variety of large data sets coming from external data sources such as social, mobile, and sensor devices. Therefore, organizations often struggle to make their data fit-for-purpose, assessing the level of trust, and ensuring data level security. According to Gartner, “Business leaders recognize that big data can help deliver better business results through valuable insights. Without an understanding of the trust implicit in the big data (and applying information trust models), organizations maybe be taking risks that undermine the value they seek.” (2)

Big Data Governance From Truth to Trust, Gartner Research Note, July 2013

“CIOs & Big Data: What Your IT Team Wants You to Know,“ – Infochimps conducted its survey of 300 IT staffers with assistance from enterprise software community site SSWUG.ORG. http://visual.ly/cios-big-data