Got no idea what Hadoop is, but think you need it? You're not alone

Open ... and Shut Hadoop is quickly becoming essential infrastructure for enterprises hoping to glean insights from the massive quantities of data they collect. The problem is that relatively few enterprises have the necessary competence to make effective use of the still-complex open-source project. While Hadoop vendors like Cloudera, Hortonworks, EMC, and MapR are doing their parts to simplify Hadoop, the real breakthrough for Hadoop may come from the applications that run on it, and not improvements to the infrastructure, according to Cloudera CEO Mike Olson.

Olson joined Hadoop heavyweights from Metamarkets, Oracle, Facebook, and MapR to discuss "The Elephant in the Enterprise: What Role Will Hadoop Play?" at a Churchill Club event earlier this week. As the panel debated the pros and cons of Hadoop, it became clear that while Hadoop is the heartbeat of the Big Data movement today, it still has a long way to go to become mainstream.

For example, Metamarkets CEO Michael Driscoll criticised Hadoop's dated batch processing in a world that increasingly demands real-time. "Hadoop is not like having a conversation with your data," he insisted. "Instead it's like having a pen pal that you write from time to time." To this Olson quipped, "To be fair to Hadoop, it's like having thousands of pen pals that write to you."

Apparently this is enough for now, with Hadoop skills topping the list of skills needed across the IT landscape, according to research firm Ovum and jobs data compiled by Indeed.com:

But how much of this is real versus media-inspired mania for the next big thing?

While Driscoll feels the need for Hadoop is real, he was also quick to caution that "we're in a Hadoop hangover" that has executives saying "I have no idea what Hadoop is but I think we need it." The problem, however, is that the reason so many executives want Hadoop (data analytics) is not what an enterprise gets out of the box. As Driscoll pointed out, "Hadoop is a technology, not a solution."

Which is why Olson is probably right when he declared, "There will be enormous Hadoop adoption, but you'll get it by virtue of the applications you run." Cloudera board member and Workday co-CEO Aneel Bhusri echoed this Olson sentiment:

In other words, Hadoop, like Linux, MySQL, PHP, MongoDB, and other open-source technology, is very important, but mostly because it will enable a rich ecosystem of applications running on top of it. Enterprises won't need to become Hadoop experts. The experts will be the SaaS providers who build Big Data applications that take advantage of Hadoop, but don't shove it in the user's face.

Driscoll's Metamarkets is set up on the premise that customers can tap into analytics as a service, with Hadoop running behind the scenes. Many enterprises, however, simply aren't yet comfortable pushing their data to the cloud, leading Oracle's SVP of Server Technologies, Andy Mendelsohn, to conclude that Hadoop appliances are a necessary intermediate step for cautious enterprises on the road to true Hadoop-based analytics-as-a-service.

In sum, Hadoop is great technology, but not yet a tool that mere mortals can manage. While Hadoop training will undoubtedly mint many more data scientists, the real value this Big Data technology promises to deliver is as a behind-the-scenes infrastructure for cloud services, including everything from Facebook to Workday to Metamarkets. While the panel participants didn't estimate the timing for an onslaught of new Big Data applications, there's enough money at stake that the answer is almost certainly: soon. Very soon. ®

Matt Asay is senior vice president of business development at Nodeable, offering systems management for managing and analysing cloud-based data. He was formerly SVP of biz dev at HTML5 start-up Strobe and chief operating officer of Ubuntu commercial operation Canonical. With more than a decade spent in open source, Asay served as Alfresco's general manager for the Americas and vice president of business development, and he helped put Novell on its open source track. Asay is an emeritus board member of the Open Source Initiative (OSI). His column, Open...and Shut, appears three times a week on The Register.