Hope springs eternal, especially in Big Data. Despite widespread failure to achieve much of anything with Big Data projects, gargantuan piles of cash keep flowing into such projects, hitting $31 billion in 2013 and expected to top $114 billion by 2018.

Yet while 60% of executives believe Big Data will upend their industries within three years—according to a recent Capgemini report—a mere 8% describe their own projects as "very successful," while another 27% call their efforts "successful."

Given how much companies are spending, one would hope for better returns. Real success, however, derives from a cultural affinity for data, and not simply a technology purchase.

Failure All The Way Down

No one seems to dispute the inherent value of data, and the more of it the better. In Capgemini's survey, fully 60% of respondents believe Big Data is going to change the world, starting with their industries.

Yet when asked about the status of their Big Data initiatives, it's clear that reality bites:

For those that have been paying attention, this isn't really news. After all, Gartner found a few years ago that while everyone was jumping into Big Data, few knew how to make it work.

As to why Big Data projects fail, the answer is "it depends." Some of the reasons are cultural ("ineffective coordination of teams across the organization"), while others are more easily fixed ("dependency on legacy systems"):

Looking at this list, it's hard to see how things improve in the short term.

Teaching Old Data Dogs New Data Tricks

Some vendors pitch a "data lake" as the solution to the first problem. Capgemini's survey finds that 79% of enterprises haven't completely integrated data sources from across the organization. To make this simpler, the data lake advocates insist that enterprises don't need to standardize data as it enters the organization; instead they can keep it in its original format and just store it in one big repository.

The fundamental issue with the data lake is that it makes certain assumptions about the users of information. It assumes that users recognize or understand the contextual bias of how data is captured, that they know how to merge and reconcile different data sources without 'a priori knowledge' and that they understand the incomplete nature of datasets, regardless of structure.

So some fail with newfangled, Hadoop-inspired data lakes, while many more fail by trying to get antiquated data infrastructure (e.g., relational databases) to fit modern data (messy, disparate, and lots of it).

The Cultural Problem

But the biggest cause of failure, even if not acknowledged as such, is that most enterprises simply don't have a culture of data-centricity. At best, they treat "Big Data" as a discrete project with a definitive completion date.

As such, they're not set up to succeed, as the Capgemini report finds:

There are many factors that go into the making of a successful Big Data implementation. However, the single biggest factor that we observed was that organizations that have a strong operating model stood apart. This operating model has multiple distinct elements, which include, among others, a well-defined organizational structure, systematic implementation plan, and strong leadership support.

Each of these three things ties into a corporate culture that appreciates and is built around data.

I would also add, following something that Zoomdata's Justin Langseth recently said to me, that design is an essential element of any successful Big Data project. The best Big Data projects will bring data to life for the rank-and-file within an enterprise, not merely the high priests and priestesses of data science.

In sum, Big Data success flows from a cultural affinity for data, which can be sparked by a strong leader within an organization, but ultimately must become how an entire company thinks about its business.

It's easy to assume that developers, employed by lines of business, have supplanted the CIO forever.

As a recent Forrester report uncovered, traditional, IT-led technology purchases are shrinking—from 55% To 47%. Still, with a mere 7.2% of tech purchases being driven solely by the line of business, something more profound is underway. That "something" is almost certainly a new wave of IT and business collaboration.

The CIO, in other words, is not dead yet.

Cats And Dogs, Together At Last

The primary driver of collaboration between IT and business is somewhat surprising. According to CapGemini, which interviewed over 1,100 CIOs and top IT decision-makers, technology is simply too important to have one group manage it:

[T]he gap between corporate ambitions and the state of the application landscape is now felt more than ever: it is no longer “just” a matter of cost and manageability, it is the indispensable role of technology as a crucial enabler for innovation, renewal and business expansion that now takes center stage. In this context, the alignment between Business and IT is a top priority.

Both groups could view technology as a competitive differentiator without actually working together. What CapGemini's survey finds, however, is that enterprises increasingly discover that it's optimal for the two groups to work together to put technology to work most effectively. The more closely the two groups are aligned, the more IT drives the technology agenda in the company, with a real focus on competitive differentiation:

This shift from cost-cutting to real innovation has liberated IT to be a real partner to the line of business, rather than "Dr. No."

Never Waste A Good Crisis

Ironically, the thing forcing IT and business together has been legacy infrastructure, which continually hobbles new application development is its near-crisis state.

Enterprises want desperately to take advantage of Big Data, mobile and other trends, but their existing infrastructure hampers their ambitions. This problem, more than anything else, has aligned the two groups, as it has forced enterprises to look seriously at replacing old-world infrastructure.

As the report finds:

Industrialization and standardization may no longer be sufficient as the pent-up demand for the next generation of applications by Business increases considerably and the pace of application landscape renewal proves to be too low.

Alternative rationalization strategies need to be considered in that case...[including] more radical strategies that aim for daring, impactful changes (e.g.,“ripping and replacing” legacy custom or ERP applications by highly standardized SaaS solutions). In the latter case, though, the transformative impact on the organization, its processes, its governance and its people should not be underestimated, the benefits can be considerable.

When IT moves too slowly, or when the two teams are poorly aligned, the business side has typically embraced the cloud rather aggressively, something ReadWrite has reported before. Therefore, while CapGemini found that three times as many CIOs are involved in initiating 35–65% of application developments than they were three years ago, when it comes to cloud services, business leads by almost 50%.

Overall, however, IT and business increasingly see each other as partners in reducing cruft and improving the enterprise application portfolio.

Make no mistake: IT and business still have a long way to go to reach deep collaboration. But we're moving in the right direction. The "us" and "them" mentality is rapidly disappearing in the face of a real need to embrace the future of data. We should expect this trend to only continue.