Data Modeling Tomorrow: Self-Describing Data Formats

The realm of IT is expanding. Enterprise applications and architecture span on-premise and cloud deployments; the latter involves both centralized and distributed computing environments. There’s also sensor data, machine-generated data, and transactional sources warranting real-time responses.

All of these factors put enormous pressure on the data modeling methods required to integrate, analyze, and derive value from information assets. Modeling must be more flexible than ever, yet still deliver its underlying functionality: consistency.

“If you look at consistency, security, and their capabilities, that has to be architected into the underlying data system,” MapR Senior Vice President of Data and Operations Jack Norris said. “That can’t be an afterthought. You’ve got to build that into the platform.”

One of the ways that organizations can ensure such consistency of data modeling with a multitude of data formats, structures, and sources is by relying on self-describing data models. These models provide the flexibility of schema needed for such a plethora of data types, but still deliver consistency in the modeling for specific applications.

Furthermore, they’re often able to provide these benefits quickly enough to keep up with, or perhaps even set, the pace of business. “That’s where we see the future going,” Norris remarked about self-describing data models.

Utilitarian Modeling
The chief advantage of self-describing data models is their utilitarian nature. Whether implemented with a JSON format or Avro (two of the most popular methods), these models are applicable to most use cases. Regardless of the type of data involved, these formats can model them according to the underlying needs of the particular applications for which they’re used. One of the main ways they provide this utility is by facilitating schema on demand, which is dictated by the underlying data. In this regard they are much more flexible than conventional relational methods of modeling, which involve creating schema for specific purposes. Any data added to a particular application or database must adhere to that schema—or modelers are required to recalibrate it to include those data. When attempting to integrate a variety of data sources, these schema limitations considerably delay time to value. Norris mentioned that, “It’s too difficult to negotiate and maintain a consistent data model across many different users and groups” partly because of the rigidity of traditional schema concerns. Self-describing data formats are much more flexible in comparison because they can derive schema on-the-fly and expedite this process.

Machine Intelligence
The drivers for self-describing data formats represent some of the most avant-garde techniques in data management. The Internet of Things has certainly made this approach much more relevant because of the copious quantities of continuously generated data involved. Applications in the industrial sector of this phenomenon, the Industrial Internet, provide a host of viable use cases in which data is generated too quickly for conventional modeling techniques. Moreover, the vast majority of such data is both unstructured and oftentimes machine-generated. According to Norris, the viability of the adaptable nature of self-describing data formats is “becoming really clear as organizations pursue more real-time applications and pursue more intelligent applications.” Forays into Artificial Intelligence are also influencing the adoption rates of self-describing data models. Oftentimes, organizations parsing through huge big data sets to determine trends or insight into a particular business problem (such as upselling opportunities) will simply have too much data from multiple sources to use traditional modeling methods. In these instances, self-describing formats are useful for allowing the data to dictate the underlying models. Norris observed that this technique is influential for organizations “harnessing machine learning, so that they have better automation and better intelligence as part of their business action.”

An Evolving Environment
Pertinent use cases for self-describing data models also include mergers, acquisitions, and other instances in which organizations have to convert a particular data format into another one. In fact, the relevance of this type of modeling extends into most facets of data management. Norris mentioned that “our streams capability, our database capability and our querying capability all support JSON natively.” The value of the adaptability of these data models is ultimately linked to the worth of big data practices in general. Conventional relational modeling methods were not designed to accommodate the multiplicity of data types which organizations regularly encounter today with big data. Self-describing data models, on the other hand, are well-suited to the rigors of the different types of schema involved in the big data world. In this respect they can offer a vital point of connection between traditional relational methods and those relevant for modeling big data. Best of all, they significantly decrease the time spent on modeling and schema concerns, creating more time for analytics and data-driven action. “The reason why big data was developed or needed in the first place is because of the volume, the variety and the velocity of data; the existing approaches no longer worked,” Norris said. “But that environment is continuing to evolve.”

Keep Calm & Get Hired!

View from @AnalyticsWeek

Login

This page is having a slideshow that uses Javascript. Your browser either doesn't support Javascript or you have it turned off. To see this page as it is meant to appear please use a Javascript enabled browser.