Hortonworks Sandbox for HDP and HDF is your chance to get started on learning, developing, testing and trying out new features. Each download comes preconfigured with interactive tutorials, sample data and developments from the Apache community.

On Tuesday, April 3, 2018, Cloudera had its quarterly earnings call, at which time the CEO, Tom Reilly. announced both disappointing earnings and a soft forward-looking outlook after which the stock has since slipped from $22.24/share to $12.95/share… a drop of a whopping 42%. According to CNBC, “The company… found that it was taking more money to acquire new customers and increase spending among existing customers.”

Is this the beginning of the end of big data? Or is it just the end of the beginning?

It isn’t as if the computing world isn’t already littered with failed distributed computing technologies. How many of you have even heard of the Distributed Computing Environment (DCE)? Or have you ever heard of CORBA (Common Object Request Broker)? Also, not so many years ago, there were lots of software vendors that had invested in providing grid management systems for helping companies implement their own distributed computing clusters, usually in the High-Performance Computing (HPC) space.

All of these technologies and approaches have more or less disappeared. The bottom line is that they all required super smart people to make them work. Or another way to put it is that they were all just too damn hard.

Then, Hadoop came along as an open-source technology. Around the same time, there was a precipitous drop in compute and storage costs and Hadoop became the distributed compute and storage platform of choice. In addition, the venture investment world went crazy with a billion dollars invested to commercialize Hadoop in an effort to ensure its success. And there is no doubt that Hadoop, and its offshoots like Spark, have far surpassed any previous attempts at making distributed computing work.

That said, the problem is that even with Hadoop and Spark, and a few billion dollars of VC money, distributed computing is still too hard. The proof point is in Cloudera’s quarterly earnings call, where they revealed they were having to spend too much money and effort to get their installed base customers deployed. That, in turn, made it difficult for customers to increase their spend with Cloudera. This only reinforces the Gartner press release, which stated that “only 15 percent of businesses reported deploying their big data project to production.”

So does that mean that Hadoop/Spark is dead? Not at all!

This time is different from before, when DCE and CORBA both died rather unceremonious deaths. In the past, the assumption was that people would get trained in universities to learn how to program and use these technologies. The assumption this time isn’t that the users will get smarter: it is that the technology will get easier. As a result, in addition to the money invested directly in Hadoop, there are also billions of dollars of venture capital money going into filling the complexity gap that was created by Hadoop's introduction.

While the first wave of technology in the big data space was simply laying the basic foundation, the second wave was investment in software vendors that provide point solutions to automate specific implementation issues for big data like managing data ingestion, doing ad-hoc data prep, handling production data pipelines, or dealing with the generation of high performance OLAP cubes. And now, the third wave of investment is happening that will enable agile data engineering by automating the end-to-end data pipeline process, from ingestion to consumption. And none of what I mention above even includes the investments by Google, Amazon, and Microsoft to simplify getting a big data environment stood up in the cloud.

The Cloudera stock drop doesn’t signal the death of Hadoop at all. What it does signal is a sign that expectations of big data companies' stock performance are finally getting right-sized until the industry delivers more cost-effective and simpler ways to get true value out of big data deployments. The good news is that the third wave of big data technologies has just arrived, so the challenges that caused last week’s negative dent in Cloudera’s stock will start to disappear. Whether that fixes Cloudera’s stock price is unknown. But it will at least accelerate time to successful production deployment from months down to days. And if Cloudera takes advantage of that next wave of technology, their stock price will, too.