Real-time needs drive data retooling

Business battles are fought in real time, and IS must keep pace. Real-time business intelligence infrastructures promise a never-ending stream of fresh information, insight and decision support to frontline knowledge workers.

Nevertheless, real-time business intelligence has not graduated to enterprise primetime yet. Most production business-intelligence implementations rely on data warehouses, which consolidate operational data loaded via scheduled batch transmissions rather than real-time updates from source databases. As a result, many organizations have rich stores of historical data in their data warehouses, but few contain information that is refreshed continuously.

A traditional data warehouse operates in store-and-forward mode, introducing latency into data delivery to reports, dashboards and other business intelligence applications. Most of today's data warehouses have been optimized for specific latency-producing operations: extraction, transformation and loading (ETL) of data from operational database management systems (DBMS); retention of that data in persistent repositories; and retrieval of that stored data into reports, graphical dashboards, multidimensional online analytical processing cubes and other business intelligence outputs.

It is possible to retool data warehouses to support real-time business intelligence. Some data warehousing vendors have begun to address these requirements in their products. Doing so requires that data warehouses -- as enterprises' master data management hubs -- be redesigned to serve also as real-time, application-layer data routers (in the broad sense of that term). For example, NCR Teradata's active data warehousing adds support for near-real-time ETL and data delivery. Just as important, the vendor has added the policy-driven event detection, processing and notification features needed to manage the flow of real-time events between data sources and consumers, as brokered through the data warehouse.

Though organizations are beginning to use active data warehouses for real-time business intelligence, no one is seriously considering deploying them as general-purpose, application-layer routers, because data warehouses usually are deployed in hub-and-spoke configurations and thus can become significant bottlenecks. Some in the industry have proposed data warehouse federation to alleviate the potential bottleneck, but most federation scenarios are fundamentally hub-and-spoke, relying on common ETL tools, metadata repositories and data staging areas.

Fortunately, other architectural approaches for real-time business intelligence are being explored. Some firms deploy an operational data store, which is similar to a data warehouse but contains only the most current consolidated data fed in through ETL tools. Another popular approach is enterprise information integration (EII), which supports real-time, federated query and update across distributed source DBMSs.

Unfortunately, there are no industry best practices for real-time business-intelligence requirements. Companies must sort through diverse approaches and try to implement them to leverage and extend their traditional, data-warehouse-based business intelligence environments.

Going forward, the data-management industry should define a clear set of real-time businessintelligence implementation best practices based on open industry standards, such as the various eventing, metadata and other interoperability specifications subsumed under the WS-* umbrella. Vendors should converge all real-time business intelligence approaches in a common service-oriented architecture framework, so that customers can deploy easily any mix of real-time, near-real-time and lagged-time business intelligence that suits their needs.

The stakes in all this are more than low-level protocol plumbing. Real-time business intelligence enables business agility through improved reporting, analysis and response to changing events. It also supports regulatory compliance. Section 409 of the Sarbanes-Oxley Act specifically mandates that companies perform real-time disclosure of material changes in their financial conditions "on a rapid and current basis."

How's that for a market driver? From a technical standpoint, how companies meet that real-time reporting requirement is entirely up to them.

Kobielus is a principal analyst at Current Analysis in Alexandria, Va. He can be reached at (703) 340-8134 or jkobielus@currentanalysis.com.

Copyright 2018 IDG Communications. ABN 14 001 592 650. All rights reserved. Reproduction in whole or in part in any form or medium without express written permission of IDG Communications is prohibited.