Data Management In The Big Data Era – Part 1

First Wave

Late 2010’s saw companies being swept by the wave of Big Data “buzz” and “promises” as how it can transform enterprises. But again, most of the companies were in wait, read and watch mode.

Mainly because, they were struggling and moving slow to identify use cases to “test”, “evaluate” Big Data ( promises ) in their context. While there were lot of news and announcements ( of success stories, magic that could happen using Big Data ), maturity of the data platform and its ecosystem was in its early days.

Second Wave

2012, from my personal experience saw little movement and interest in the space.

Platform providers figured that unless they lead the way to provide tangible, relatable, real world success stories, equally call out the limitations and rough edges and provide enough industry reference points on use cases and what Big Data entails – they’d perish into history. They kept pushing the narrative.

However from an adoption stand point, industry at large was titled heavily between Internet and E-Commerce companies and the rest.

Internet and E-Commerce companies were grappling with issues to manage data – humongous amount of data – all the while wanting to save money and there by invested early on Data Platforms ( in spite of maturity of the platforms ).

Rest of the industry – Banks and Financial Services led the pack, with Health Care, Pharmaceuticals following – were moving, but cautiously. They had to balance between IT investments made vs. meeting Regulatory and Compliance needs vs. venturing into a new space, investing in new ecosystem.

Big Data was directly gunning to question and change, their status quo !

Second wave continued with force through late 2014 (approximately), saw lot of flashy announcements, but gingerly movement. But in all, enterprise were looking at each other for their first move and success/failure stories to learn and make a move. But they couldn’t stop the heavy influence of Big Data.

As with legacy, while technology platforms and innovations were largely industry agnostic – every industry segment had its own nuances that industry at large was struggling to see into the future, with Big Data in the equation and ecosystem.

Fear Of Missing Out The “Unknown”

At the time, much of the focus was toward a “thorough” evaluation of use cases, leading to “smaller” investments in Big Data. Not to forget, the talent pool ( at the time ) was scarce to justify making larger investments.

Such small investments did move the needle for the companies, but it was mostly a “trial” period of a “nascent revolution”. Pressure for enterprises were if you didn’t have a program or a strategy on Big Data, you’d miss out a revolution ( that everybody talked about, read about ), but miss out something that stood “largely undefined”.

Blind Spots

Choices for the innovators ( platform providers, tool developers ) at the time were enormous, that much of the focus was to build or enhance “distributed data platforms” and tools and utilities that could “work with them” to keep the processes running and moving.

Fitment of such platforms or solutions ( at the time ) from an enterprise architecture stand point was “slotted for future release”, lacking much of thought or ideas.

But that was the inevitable wave that was going to sweep, for ultimately the problems that enterprises were confronting – with legacy, the tools and processes that were put in place ( for ages ) to help tackle them – were not going to be “automatically change and be usable” with new tools and techniques that Big Data platforms and tools brought in.