Think Managing Big Data Is Much Too Complex? Just Wait

The complexity of big data, and the learning curve for data professionals charged with managing it, continues to grow, much to the frustration of many organizations. At the recent Strata & Hadoop World conference in New York, Information Management spoke with Jason Schroedl, BlueData’s vice president of marketing, about the implications.

Information Management: What are the most common themes that you heard from attendees?

Jason Schroedl: Probably the most common theme is that big data today is about much more than just Hadoop. It’s about Hadoop and Spark, Kafka, Flink, TensorFlow, NiFi, etc.; about NoSQL databases like Cassandra, MemSQL, MongoDB, etc. and about data science tools like R, Python, Anaconda, H2O, Zeppelin, Jupyter, etc.; and all the BI / ETL / visualization / analytics applications that were featured at the Strata + Hadoop World event.

This aligns with what we’ve been seeing with dozens of enterprise customers over the past year, and it came through loud and clear during our meetings with participants at the conference.

The ecosystem continues to evolve and expand at an astounding pace … and the success or failure of a big data implementation may hinge on how well the organization handles the rapidly changing menagerie of applications and tools that their data scientists, developers, analysts, and engineers want to use.

IM: What are the most common data challenges that attendees are facing?

Schroedl: Finding the right set of big data expertise and implementing best practices for this continuously evolving and expanding ecosystem of big data applications and tools is both difficult and time consuming.

It’s even more difficult due to the rapid pace of big data innovation, as new versions and new tools are constantly being released (and of course, data science teams want to use the latest and greatest). This continues to be one of the biggest challenges for many enterprise organizations: the complexity and learning curve for big data deployments is daunting, and it’s not getting any easier.