I had an inquiry today from a client using packaged software for a business system that is built on a proprietary, non-relational datastore (in this case an object-oriented DBMS.) They have an older version of the product – having “failed” with a recent upgrade attempt.

The client contacted me to ask about ways to integrate this OODBMS-based system with others in their environment. They said the vendor-provided utilities were not very good and hard to use, and the vendor has not given them any confidence it will improve. The few staff programmers who have learned enough internals have already built a number of one-off connections using multiple methods, and were looking for a more generalizable way to create a layer for other systems to use when they need data from the underlying database. They expect more such requests, and foresee chaos, challenges hiring and retaining people with the right skills, and cycles of increasing cost and operational complexity.

My reply: “you’re absolutely right.”

Their only recourse, absent utilities that can extract data when needed (via federation, virtualization or distributed processing models as identified in Gartner’s Information Capabilities Framework), is to build and materialize an intermediate datastore – ODS- or DW-style. That means an exhaustive effort to anticipate all likely future requests for data. It’s unlikely they will be able to do so, and any change to that system will be as difficult as the one-offs they build now. They could literally recreate the entire database in an alternative architecture that their other systems can access via conventional methods, but the cost of building and maintaining such a redundant “layer” might rival replacement cost.

Thus, my conclusion: their vendor has demonstrated itself to be an architectural cul-de-sac they might want to consider exiting. If you have any similar systems in your portfolio, here’s a New Years’ resolution for you: back out of that dead end. As soon as you can.

Merv Adrian is an analyst following database and adjacent technologies as extreme data transforms assumptions about what to persist as well as when, where and how. He also watches the way the software/hardware boundary… Read Full Bio

Great topic and it’s something we see all the time. These older, operational systems truly represent the VARIETY challenges of Big Data. It’s a common theme…the source data is diverse, and doesn’t fall into neat relational structures for analytics.

Today, our customers routinely access this data (and data from outside the walls of the company) by transforming the operational reports that come out of these applications. These reports are a business critical information asset that have been tested and audited for accuracy, especially in public companies.

By transforming the report data (from file) and using it as a data source, our users usurp the complexities described in your post, and get immediate gratification from the report and the “point in time” calculations that only exist in the report itself.

@Merv, if there was a set of open (or at least, published) APIs, of any kind, to access either the app’s objects, or the underlying database, a custom connector could be built for a data integration tool that would then provide an easier way to connect to any other system. Since they won’t be upgrading, maintenance is probably not an issue there. But if the system is entirely closed and locked, exiting that dead-end seems to be the only alternative.

I have been watching these cases on various occasions. I call them “Data revolution for data proletarians.” Many of the proletarians are not Gartner clients so we cannot help them, moreover, they disseminate worst practices. It is about time to step back and look at good old principles not as legacy but as heritage. And, BTW, they are not so old.

About

Comments or opinions expressed on this blog are those of the individual contributors only, and do not necessarily represent the views of Gartner, Inc. or its management. Readers may copy and redistribute blog postings on other blogs, or otherwise for private, non-commercial or journalistic purposes, with attribution to Gartner. This content may not be used for any other purposes in any other formats or media. The content on this blog is provided on an "as-is" basis. Gartner shall not be liable for any damages whatsoever arising out of the content or use of this blog.