Beginning with the ‘Data Lake’ concept, Big Data and Hadoop have often earned a reputation as a place where data goes, not where data comes from. IT drives the economies of infrastructure; business end-users drive the value of data with ‘self-service’. Divergence or convergence?

Goodman considers himself an advocate “the last mile of data”. For BI before big data and since, the challenge remains the same: “getting the right information to the right person at the right time.” No surprise: Gartner defines Modern BI as anchored in self service. The need has not become less intense:

…there’s no denying the need to move faster, to get information created or transitioned from large volumes of data into creating consumable information. That need is there and it’s only accelerating. So we’re at a kind of a crossroads as an industry.

Hadoop in IT: shadow of cloud?

Even if Big Data can make the struggle harder, he says, the complexity of Hadoop means IT is far from irrelevant:

[L]egacy BI systems, legacy analytics systems, were [also] fairly complex. For a very long time IT was the gatekeeper and, potentially, the order taker for getting data into the hands of the people who need it. issues with governance and quality, and same version of the truth, and those are also real issues that do occur in a decentralized approach.

Great ‘news’: IT is from Mars and Business users are from Venus. Now we’re in trouble … or are we? IT wants to make sure the data is well cared for, while business is out pursuing competitive advantage?

There’s useful lesson here from the world of cloud-native computing–and I don’t mean ‘cloud-hosted visualization’, but the fundamentals of cloud architecture. The roots of cloud architecture were in virtualization. IT managers pounced on it: same applications, same business engagement, less footprint.

But to its detriment, IT drove the transition to infrastructure virtualization by consolidating existing server infrastructure, without confronting the need for change on two fronts: unchanged application architecture and business engagement was a bug, not a feature. IT cut costs with virtual servers on hypervisors, while business users spawned users ‘shadow IT’ and took applications to cloud platforms to pursue new markets and competitive advantage.

Today, no one questions the value of transitioning to a cloud architecture, and the early movers to cloud are winners. As former EMC CTO Mark Lewis put it in an interview this week, “You don’t merge Sears and JC Penney and get Amazon.” Lead or be lunch. So, back to Ryan Goodman, to see how the cautionary cloud holds lessons for BI on Big data.

One of the challenges that we’ve seen with Hadoop or any Big Data technology is that because it was looked at as a technology investment. … They kinda buy into the hype (but), not necessarily extract the kind of value that they were thinking, just because they can access or keep all of their data and supposedly process and extract insights from it.

Visualize a new kind of self service BI

Yet it is possible, Goodman suggests, to literally visualize your way out of the trap.

If you can bridge the gap in understanding, which visualization is so amazing at doing, it’s much easier for people to comprehend charts and pictures because their brain is so great at pattern recognition but not necessarily at picking up patterns we’re just staring at, columns, rows, in a spreadsheet. When you’re able to visualize this and build that lightweight, interactive business application that you’re talking about on top of your Hadoop cluster, now you can actually close that gap.

What does this mean for the organizations, the Mars/Venus split of IT/Business? Self-service BI does not need to create a new form of Shadow IT. BI industry guru Wayne Eckerson has the right idea, says Goodman:

Eckerson … has a very interesting concept of a federated approach … taking folks from IT and essentially injecting them into the business, to become more or less the eyes and ears back to the BI competency or analytics competency center, or Big Data competency center … you have this notion of cross pollination where you’re taking members from one part of the organization and injecting them into to the other.

The answer reconciling Data Offense and Data Defense is this: make sure everyone is on the same team.