Federalism in Data Management Series: No. 3

“We worked long and hard to get to a single version of the truth. Then we realized that we can’t handle a single version of the truth,” said a data architect at a global manufacturing company. This company built a beautifully architected and perfectly conformed enterprise data warehouse with a single unified portal for BI and reporting. But business units everywhere, from Europe to Latin America, expressed great frustration because they can only get information out of the system in one way. As a result, spread-marts proliferated.

Every enterprise BI initiative faces the fundamental conflict between the need for local autonomy and flexibility, and the need for standardization and economy of scale. There’re indisputable benefits to a single-source, multi-purpose system that holds the “single version of the Truth”. However, a large enterprise is a complex, multiple-layered, interrelated, and dynamic collection of business units and functions spread over continents with very different business climates. Local business units have legitimate needs for many variations of the “Truth”, and for agility, they want to be in control of these variations. Plus, there’re always local data needs not shared by other business units. BI, perhaps more than other data-related initiatives, is particularly sensitive to local requirements.

Let’s consider the two extremes solutions. At one extreme, we build a gigantic data warehouse managed by an enterprise team. With no exception, all BI needs everywhere are serviced with data from this data warehouse. At the other extreme, we have completely separate “stovepipe” data warehouses for each region or business areas and give them complete autonomy. For most large organizations, these extreme solutions are not sustainable. They’re the technological equivalents of Authoritarianism and Anarchy. As the current unrest in Egypt shows, the two are not as far apart as we like to think. Spread-marts are often caused by too much central control on data.

The principles of federalism can guide us to an optimal solution to this conflict. Recall that federalism means the central authority should have control only over a limited and clearly defined set of the things. The rest would be locally controlled. It strikes a balance between the two extremes by drawing a clear boundary for what is considered centralized. For BI, this means the central, enterprise level authority should control a limited, core data model and a core set of data, while allowing local business units to extend the core anyway they like to fit their needs, as long as they don’t change the core.

Technically, a federation has a central data warehouse and a group of member data warehouses. In the most common design pattern, the central data warehouse distributes a core data model and global master and reference data to the federation members. The members are free to extend them to address local needs, but are required to adhere to the core to enable a level of central control and standardization. Meanwhile, transactions data flow back up from federation member instances to the central instance based on the aggregation rules inherent in the core model. As a result, the central data warehouse can offer cross-functional and/or cross-regional views of the enterprise as whole. (Keep in mind that federalism is a data governance model that lays out responsibility and decision rights, not a physical architecture. All these data warehouses can very well live on the same physical database.)

Many Kalido customers, including Shell, BP and Unilever, built global BI programs based on federalist principles with great success.

In my next blog, I’ll talk about how federalism can guide data policy management.

5 Responses to “Can You Handle a Single Version of the Truth?”

Our company is almost completely run by “spreadmarts” and I don’t think we are ready for a single version of the truth. This article has great insights on a workable path forward. Thanks as always for the well stated insights!

Trackbacks/Pingbacks

[...] This post was mentioned on Twitter by Kalido and Scott Hartz, Winston Chen. Winston Chen said: My new blog: "Can you handle a single version of the truth?" http://bit.ly/hTlx6U #BusinessIntelligence #DataGovernance #BI [...]