Big Data Meets Trough Of Disillusionment: Gartner

Gartner says cloud, in-memory databases are headed into Hype Cycle "trough of disillusionment." Productivity expected within two to five years.

Cloud computing and in-memory databases, two darlings of the big data movement, have passed the "peak of inflated expectations" and are headed into the "trough of disillusionment," according to Gartner's Hype Cycle for Big Data, 2013.

Hype Cycles are Gartner's way of communicating the degree of hyperbole versus productivity associated with emerging technologies, with the peak and trough being followed by a "slope of enlightenment" and progressing to "plateau or productivity" as technologies mature. In the case of cloud and in-memory databases, the plateau is two to five years away, according to research discussed at last month's Gartner Symposium.

In the case of cloud, compute capacity and storage in the cloud are thought of by many as cost-saving options, but "organizations are a little disappointed about perceived cost versus reality," Gartner analyst Nick Heudecker told InformationWeek. "We're starting to see some pullback with organizations realizing that they may not be saving that much money."

That could be because moving anything approaching big data scale into the cloud involves storage and data-movement costs that can add up. Heudecker cited the case of the streaming service Spotify, which recently opted to move its cloud deployment on-premises.

"Cloud makes sense when you can't predict your load or demand, but once you have a very predictable workload and you can easily estimate what kind of resources you're going to need, the cloud may not be as attractive," Heudecker said.

Spotify's example matches the experience of Rob Rosen, big data field lead at Pentaho, who said he often sees cloud as the place where big data practitioners start. "The cloud is a great place to do prototyping and early architecture work, but once they succeed and start to scale up, they start to see monthly bills that make them say, 'hey, wait a minute; maybe it's time to bring this in house,'" Rosen said.

On the in-memory database front, Heudecker said the technology has captured a lot of attention, but now questions are emerging as people look beyond the touted benefits of fast performance, simplified infrastructure, and potentially smaller data footprints and power requirements. The questions are popping up about support for disaster recovery and high availability, availability of database tooling, and the impact on application development teams.

"Right now in-memory technology is adolescent," said Heudecker. "We forecast that it will be two to five years before it reaches the plateau of productivity -- perhaps longer, depending on the database."

SAP's Hana platform is perhaps the best known in-memory database platform, but others include the Aerospike database, in use for several years mostly in online advertising; IBM's BLU acceleration feature for DB2, introduced earlier this year; and Microsoft SQL Server 2014, an update set for release in the first half of next year that will include an In-Memory Online Transaction Processing option that's now in public beta. Oracle preannounced an in-memory option for Oracle database, but beta and GA release dates have not been announced.

Other technologies plotted on Gartner's Hype Cycle for Big Data include Hadoop SQL interfaces and the Internet of Things now headed for the Peak; social analytics and Hadoop distributions headed into the Trough; and social media monitoring and predictive analytics now on the Plateau. One final observation from Heudecker is that the Big Data Hype Cycle is going to be particularly dynamic.

"There are a large number of technologies on this Hype Cycle, and the majority of them are clustered around the Peak," he said. "That means we're going to see a lot of change over the next two to five years."

Emerging software tools now make analytics feasible -- and cost-effective -- for most companies. Also in the Brave The Big Data Wave issue of InformationWeek: Have doubts about NoSQL consistency? Meet Kyle Kingsbury's Call Me Maybe project. (Free registration required.)

SAP is the self-proclaimed leader on in-memory technology, but it has to show it's leading out there in the real world. Let's start hearing more concrete customer success stories and less hype about unspecified "never-before-possible applications" unmeasured "dramatically faster performance" and unverifiable "transformation without disruption." I'm talking to customers and hearing lots of good things, but there are realities (like the cost of new servers) and tradeoffs (like rework required to change apps and analytics) that SAP doesn't acknowledge.

Customers need to know they will get to that "plateau of productivity." Hype will deepen the dissolusionment while real-world case examples will help establish realistic expectations.

You're absolutely right about Big Data being in the "trough of disillusionment". It is, and for good reason: people have been burned by the solutions they've tried.

What's been lacking up to this point is a cohesive data strategy. There needs to be a plan for asking questions, getting answers from the data, and then empowering executives to make data-driven decisions based on that.

I think that the idea that ALL technology has this trough is misleading. However; This time definitely comes where the behaviour moves beyond raw hype. And we do seem to like a bit of hype (Think IOT!)

In the #bigdata world that time (what they call the trough) has clearly come - and for me as an experienced Pentaho consultant the time is that of genuine implementation. People are now doing REAL bigdata projects, not POCs and extracting real value from the tools. The tools have matured (Hadoop and Pentaho), are now very usable and do a good job.

Obviously not everyone needs it. But for those that do now is a far more interesting time in bigdata than it was a year ago.

To learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.

Chances are your organization is adopting cloud computing in one way or another -- or in multiple ways. Understanding the skills you need and how cloud affects IT operations and networking will help you adapt.