Rob Karel's Blog

The integration software market is a busy place, with technologies such as ETL (extract, transform, and load), ESB (enterprise service bus), CDC (change data capture) and IC-BPMS (integration-centric business process management suites) crowding the landscape. The technologies within this "integration acronym buffet" address a common requirement: they either physically or virtually deliver information from point A to point B with integrity. (For more information on the variety of integration options available, see "2009 Update: Evaluating Integration Alternatives.") Some approaches, such as ETL and ESB, have developed into multi-billion dollar stand-alone markets. But other integration alternatives, while valuable, haven't survived as well as a standalone market segment. EII (enterprise information integration) is one example of this.

In a nutshell, EII -- also often referred to as data federation -- is an umbrella term that arches over a collection of technologies and best practices for providing custom views into multiple data sources as a way of integrating data and content for real-time read and write access by applications. Though integration professionals have been using EII for many years in niche applications, it has struggled to find the ideal scenarios in which it has a clear advantage over other integration approaches.

Where enterprise data warehousing (EDW) is concerned, EII provides an on-demand, decentralized "virtual" alternative, but has not dislodged ETL's role in the centralized, batch-oriented EDW architectures that predominate throughout the corporate world. For real-time and operational business intelligence (BI) requirements, EII supports low-latency queries across distributed data environments, but it competes against other approaches such as trickle-feed ETL and CDC. And for real-time read/write access across disparate databases, EII competes against more versatile approaches such as ESB and information as a service (IaaS).

In more recent years, EII has found adoption as an adjunct to enterprises' core EDW, BI, and transactional computing integration strategies--not as the main approach. When they implement EII, information architects generally use this technique sparingly, and primarily for tactical, project-based requirements -- not as an enterprise standard. So it’s no surprise that a stand-alone EII tools marketplace has failed to develop, or that most EII vendors have slowly disappeared via acquisition into larger BI, EDW, data integration, or database platforms such as Business Objects's, Red Hat's, and Sybase's acquisitions of Medience, MetaMatrix, and Avaki, respectively. Meanwhile, the remaining EII pure-play vendors such as Composite Software have wisely shifted their solutions and messaging towards a broader focus on SOA (service-oriented architecture) and IaaS.

Ironically, as the EII-branded market continues to consolidate and merge with other data management markets and essentially disappear as a recognized standalone segment, EII's earliest goal of enabling virtual data warehousing may now be poised to become reality. As the complexities of today's business operations increasingly requires real-time decision making, real-time data warehousing options are once again being discussing by data architects across industries. As Forrester analyst James Kobielus discussed in his research, "Federation: Sharpen Your Focus On Vast Constellations Of Data," multiple use cases have arisen where leveraging EII tools and techniques may be the best way to deliver information insights in a complex, heterogeneous environment. And federated use cases go beyond data warehousing and business intelligence. Architects designing solutions to manage complex enterprise content management (ECM) and master data management (MDM) strategies also consider federation as a legitimate means to bridge information siloes.

Does this mean the standalone EII market is ready for a comeback? Not quite. What it does mean is that EII may have finally found its place in the software world. Vendors building data management solutions focusing on areas such as IaaS, SOA, BI, data integration, data warehousing, and MDM should continue to invest in seamlessly integrating EII-like federation capabilities as an embedded integration technique available to their customers. An executive at a data integration software company recently shared with us his perception of how their customers have evolved to think about EII: "What was once called EII are really just a set of integrated access, abstraction, federation, and delivery capabilities behind (a) Virtual Data Federation as a complementary data integration tool in the DI project toolbox and (b) IaaS as an enterprise data virtualization architecture approach for the enterprise." So integration architects should take a step back and consider all of the options within the integration acronym buffet not as competing tools, but complementary techniques that must build the foundation for your organization’s cross-enterprise integration competency.

Comments

I greatly disagree with your assessment of EII. As author of EII: A Pragmatic Approach, EII is not about data federation, but about semantics. It's about representing data in multiple contexts and managing the relationship between the data and those contexts. This is a very, very complex problem domain and we've barely scratched the surface of it.The tools you mention were really data mashup and data federation tools, not EII tools. That was one problem. The tools didn't meet the requirements for the problem domain. The 2nd major issue is that there is a very expensive data quality issue that must be dealt with before EII can occur.Having successfully implemented EII twice at 2 major corporations, I have seen the value it can bring to the business with regard to converting data into information and information into knowledge.JP

JP - Thanks very much for your comments. I actually think you and I are in agreement, but ironically the problem with EII is not necessarily the technology - it's the branding. The data mashup/federation technology as you described them may not be what you today consider EII - but many vendors messaged these tools as EII to the marketplace and as you correctly pointed out, these tools did not deliver on their promises - especially in areas like data quality and flexibility.As I mentioned, I completely agree there are high value strategic applications for creating a vitualized semantic abstraction layer of critical information, be it through an Information Fabric or Information-as-a-Service architecture or other approaches. My caution to integration architects is not to assume that these approaches offer a nirvana cure-all for all data integration requirements.In addition to data integration technologies, I also cover data quality, master data management, and data governance best practices - and my primary concern for any organization looking to provide a 'single version of truth' - be it physically stored in a data warehouse or virtualized in a semantic abstraction - is that the business must first define what they consider the truth. And then the supporting data must be standardized, reconciled, and validated against that truth. Only then, as you pointed out, can EII - or any other integration technique - deliver that truth in any context.You and I will not be the only ones to debate terminology because that is the unfortunate state of this market, but on the positive side discussions such as thse will hopefully help IT decision makers to know what questions to ask next.Thanks again for sharing your thoughts.