#datavault Models, Business Purpose, Data as an Asset

This entry takes a step far back in time, to when I created the Data Vault model. I will share with you the thoughts around it’s creation, it’s intention and it’s original purpose. The reason why this entry is here, is because I’ve begun hearing that: “the data vault model is not business focused” – and I want to put that notion to bed, immediately – because it simply isn’t true. If this is what you believe about the data vault model, then a) you’re not building it right, b) you don’t fully understand it’s purpose c) you might never have taken the time to take an authorized training course from me or one of my authorized trainers.

Data as an Asset on the Books…

One of the original intents of the data vault model was and still is, to provide value to the enterprise – in terms of data as an asset on the books. Something that can be assigned a monetary value, and then depreciated over time. Now, how that is done is a very very difficult task (actually assigning a dollar value). I will leave those thoughts for another blog post. Returning to the original premise…

Hypothesis: In order for raw data to have value, it must: a) be tied directly to business processes, b) must contain enough completeness so that when a human looks at the data, they can make decisions about how to turn that data in to information (make it available to business decision making). After all, this is where perceived value comes from – when the impact of the business decision is felt and measured against the profitability and overhead curves of the business.

Supporting the Hypothesis…

This means, in order to tie data directly to business processes, it must be linked (in some fashion) to business keys. Why?

Imagine a Ferrari, that at the dealer has a value (sticker price) of $245,000. You purchase the car, and the dealer says: oh, wait, we lost the key. What is the value of the car to YOU as an official and registered owner of the car, while the key is lost?

Yep, ZERO.

Once the key is found, and you drive the car home, the value of the car WITH THE KEY (while it’s still on the dealers lot) is the purchase price of the car.

Business keys are important, very important…

Business keys are the unique identifiers which the business people use to find, assign, modify, create and enrich descriptive data to. Business keys are the unique identifiers which help the business people and the machines / applications tie the data together, and track it horizontally through the business (from sales to finance to contracts to manufacturing for instance). Without business keys, the value of the data itself is drum roll please… ZERO.

By the way, all of this and more is taught in any of my authorized CDVP2 (Certified Data Vault 2.0 Practitioner) courses)- you will not get this from anywhere else! Check out more at: http://LearnDataVault.com

Business keys are everywhere…

Ever had a bank loan? What was the loan Number? How about a wireless phone? what was the phone number? How about a water or electricity bill? what was your customer number? What about a contract with someone, what was the contract number?

You don’t have to look very far in the real world to find your business keys. The very companies you work for use them to track associations of data directly to you. Now, what’s the value of you as a customer for one of these companies? The answer? it varies over time – depending on how much money you invest in them, or how much money you pay them monthly or over the life of the contract. But that’s just the measurable value. Then, there’s the intrinsic value – because if you’ve spent money with them before, then chances are pretty good (unless something bad happens) that you will spend money with them again.

However, if you sign up with them again, you will most likely be assigned a new business key. This key will be used to track your information back through the business processes.

Ok, fine. What’s that got to do with Data Vault?

The Data Vault model in it’s original form, is built to track business keys and their surrounding context through the lifecycle of business processes. The business processes are executed by “source system applications” – they constitute the reality (not the perception of how the business should work) – but the reality of how the business actually IS working. So that said, we arrive at our first construct: the Hub.

The original Hub in the Data Vault model (when built properly, and not containing degenerate or weak keys, as some out there on the internet would have you believe).. ACTUALLY WAS JUST THE BUSINESS KEY. You heard me right, no load date, no record source, no primary key (surrogate OR HASH). It actually was, a purely all natural original business key value. The Hub was JUST that… A HUB (a unique list of business keys).

Now, you think: well, there’s all this debate around surrogate vs hash keys, and debate around how to use the load date, and how to use the record source. Let me stop you right there – NONE of these fields hold business value except: when it comes time to troubleshoot the arriving nature of the data.

That said, the original Satellite had a “replication” of the Hubs natural business key, combined with a Load Date – this had to be done in order to provide history, or data over time. Again, the original satellite definition had NO record source, NO load end date (which are dead now anyway), NO current record indicator, or anything else… Just the key structure plus the descriptive data that arrived on the source feeds.

Which brings me to the Link. The original link had (again) a replication of JUST the business keys from two or more parent Hubs. NO Load Date, NO record source, NO Hash, No Surrogate.

Well, why is that important?

Because folks are now arguing with me, and with others over “hash key versus surrogate key” – when there is no true business value there, and it is not related in any way to the original purpose of Data Vault modeling.

Why are Hashes or Surrogates used as primary keys then?

ONE WORD ANSWER: Performance. In reality, the surrogates (IF you still use them or insist on using them) provide join performance – especially if they are clustered on NON-MPP solutions. On MPP, they really don’t matter, as the internal optimizer changes the values over to a different representation to find the data on a node/AMP/module, etc.. Hashes, on the other hand (as explained previously) are there to solve heterogeneous cross-platform load performance bottlenecks – eliminating all the caching lookups that take place. I refuse to get in to the technical details any further on this blog entry. Search my blog for many many posts about hashes, sequences, joins, bottlenecks, etc…

The point is: the original model never had these “technical performance components anyway” – so why are we still arguing over what to use? We are losing precious valuable time when we should instead be focused on solving business problems.

So now tell me, what’s this got to do with Data as an Asset?

Right, back to the business. The original Data Vault Model is positioned to help categorize and place in to a basic flattened hierarchy, business keys – and their surrounding context. The original Data Vault model provides pure business value by demonstrating the gaps between the business perception (of the way they THINK their business is running) and the reality (the way the data is actually captured and processed through business processes and systems).

Wait a minute… you’re telling me there’s gold in the raw data?

YEP YEP YEP… Especially when you build the correct Data Vault Model. If you build a source system Data Vault Model, the value of the solution drops to one tenth of one percent overall. Integrate your business keys in your hubs – properly to achieve maximum asset valuation.

What’s the best way to build a Data Vault Model?

Don’t start with the Data Vault Model, start with an ontology of business terms. A properly built ontology of business terms (when taken to the right level of grain) can and should identify: (you guessed it!!) BUSINESS KEYS AND THEIR RELATIONSHIPS. A properly built ontology can even be assigned business estimated intrinsic value (ie: how much money would we lose if we had this key blank?, how much money will it cost us to have duplicates? how much money will it cost us if the data is incomplete or wrong?) These valuations can be assigned, (and yes they need to be maintained through good data stewardship, and data governance. But… You can take a properly built ontology and generate the right Data Vault Model. Yes, Automation and generation.

This, leads ultimately to tying direct dollar figures to the data sets through a proper ontology and data governance strategy. This is something that one of my earliest customers: QSuper in Australia has been doing for years, all on Data Vault 2.0, and yes – with hash keys. This, is something that can make sense to the business.

At the end of the day, you still should be constructing some form of a business based output layer. Should it be a Business Vault? Maybe – depends on the case. It some cases today (more and more frequently) we are constructing virtual business views (virtual marts) directly on top of the raw Data Vault. And if you’re unsure about this part (performance wise), then read up on Point-in-time and Bridge tables – the two most misunderstood, and misused, and misapplied modeling techniques in the Data Vault landscape today.

But I digress… The summary of it all please..

Here are my points:

Data Vault modeling NEVER was about the “source system”, NEVER was about the sequences vs hash key battles

Data Vault modeling was, is and always will be ABOUT THE BUSINESS. And if the Data Vault you have in place today is not currently about the business, then unfortunately you’ve hired the wrong people, and those people need to go back to school and re-learn what Data Vault really means. OR you’ve built the wrong solution, and you need to fix it – immediately.

There is value in Raw Data – when integrated by business keys. Think Gap Analysis!

There is a need to understand truly what a Hub IS and IS NOT, truly what a Link Is and IS NOT, same with the Satellite. A Link CAN NEVER be a Hub – sorry, that’s the way it is.

There is a need to tie data as an asset back to the business, this is done through the business keys.

Ontologies are a very very important asset to the corporation – if built at the enterprise level, you must focus on ontologies while you are building the Data Vault solution, or the full value of the raw Data Vault cannot be realized.

There will be more on these subjects as we go forward.

Thank-you for your consideration, as always, I’m open to your thoughts.

(C) Copyright Dan Linstedt 2016 all rights reserved.

PS: if you want to offer a negative comment, or a differing viewpoint, that’s fine – but please don’t hide behind anonymous emails, don’t be afraid to tell the world who you really are, and take a stand by golly.

2 Responses to “#datavault Models, Business Purpose, Data as an Asset”

Thanks Dan, i am working on Data Vault projects for three years in very complex systems (no typical HR application) and i noticed somethings. I would like to share them with you:

* At the start the D.V. Model looks good. but the Model starts to explode after many Changes in Source Systems, because of too many new Sataliets. We still have good performance, because we use only INSERT. But the Model getting larger and larger and It is getting hard to keep modeling in D.V. It costs us time during the Modelling, sometimes we open discussions like for example splitting Sataliets ,building new BRIDGES/PITS tables…etc

*. While we using D.V. to build our DataMarts and other SoftRules i hear the following alot from different people in different projects, “I wish that our Stage-Layer is Core-Layer, its really easier to query and to understand”

From my point of view, it is really not a bad idea, that we build a Core-layer that just looks like Sources without modelling, i see the following advantages:

1. We don’t need to Model the Core, and all those time discussions about different Core-Modelling issues are not needed. -> We are Faster !
2. No Normalization in Core ! Number of Tables from Sources = Number of Tables in Core -> Queries are faster (fewer joins).
3. When we need to understand the business and processes behind in Source System, we usualy be in this situation where we need to understand the Model of the Source System. And could happens we start to build a queries on Source Systems to understand the Data and Business. Since the Core looks exactly like sources, this would makes lifer easier to make Business Rules and build the DataMarts.
4. We still can you same techniques from D.V.: HashVaules to find the capture the new Changes, Only Insert.
5. Loading the data in Core would be faster, because number of Tables are fewer.
6. One Generic Job that loads the data from Source the Stage, and then from Stage to Core .
7. We don’t need to do the changes to MetaData manually -> Automatisation. We read the MetaData from SourceSystems and we apply it to Core.

I see here, that we can get more faster, more automatisation and the best part that we let our experts in Data Warehouse to focus more on Building and understanding Business, and they would be relieved from the Core of Data Warehouse.

If the model is exploding, then perhaps it is not modeled correctly. It may require an assessment and refinement to correct what is happening.

If you wish to use a fully historical persistent stage core layer, then be my guest. Just realize that the fully persistent staging core layer is lacking the following pieces: Delta tracking, Integration by business key, relationship and hierarchy data, and master data capabilities.

I’m not here to tell you what works best for your organization, all I can do is point out the different failings and potential bottlenecks / roadblocks of proposed methods. I would suggest your organization might benefit from an on-site visit, and a full review of the solution being proposed, as well as your currently exploded Data Vault.

It sounds to me as if what was constructed was “source system data vault models” – one per source system, rather than the actual purpose-driven Data Warehouse at the enterprise level which is the real directive of proper Data Vault Modeling.