Data maturity – react, inform, predict & transform

In 2010, The Economist published a special report “Data, data everywhere”[1]. 2010 is a life time ago in technology terms – that’s the year Apple released the iPhone 4, the first with a front-facing camera and perhaps inadvertently sparked the Selfie revolution – and the article reeled at how the rate of growth of data we generated, stored and processed was exponential.

At this point Sky Telescope Surveys were storing 140TB of data, but new ones were planned that would generate that much data every few days, Walmart were entering the next order of magnitude and popularising the term peta-byte.

For me the interesting underlying message within the report was that whilst the technologies were there to handle this volume of data, the processes and techniques for actually extracting value from all that data were problematic.

Winding the clock forward the best part of a decade and the proliferation of big data (or, in reality we’re now rightfully dropping the ‘big’ label in many places) has exploded, it’s on every businesses technical radar somewhere, but the challenges from 2010 still abound.

The tools available are light years ahead. Cloud vendors make provisioning the infrastructure required to work at this scale, once the prevue of weeks or months of fine tuning by gurus and specialists, a routine affair achievable in hours, but still there are challenges.

The key to understanding this is that the technology itself is not the be all and end all. How we use it, how we model it, how we secure, govern and maintain it are the key to unlocking the value it contains and this brings me onto an important strategic and management consideration when looking at data and planning a project to leverage it in organisations of any size – ironically it’s now much easier the smaller your organisation is.

The key to successfully leveraging your data is not to run before you walk. It’s an old adage and applies to virtually all walks of life, but it’s so oft forgotten. The key to this is to understand the data maturity of your organisation and to always be focusing on improving, moving perhaps towards the next level if there’s value to be found there – or not where it makes no commercial sense – but you can’t simply jump in at the end.

The major clusters of data maturity centre around these four key pillars (with anecdotes):

Reactive – Data is used reactively – you look at the month end report and think oh, that’s what happened

Informative – Data informs your decision making – data is tracked and embedded in regular processes so you can see what’s happening right now and can decide to act based upon facts

Predictive – Your current and past data is used and modelled to help predict and project what will happen allowing you to play scenarios and optimise

Transformative – You close the feedback look – your continuously adapt your business processes, your products and your relationships with customers based upon the insights, trends & insights gleaned from your data which helps you identify opportunities, shore up weaknesses and cleave dead weight

This nirvana will look very different for many organisations – as indeed it should, each company’s data will inform different opportunities, adaptations and the strategies they choose to capitalise on those opportunities may be very different.

As your business moves towards learning to run it needs to build the culture – the organisational muscle-memory – for how to control the quality, the evolution of the models, the processes & political will to adapt your business based upon what your data is telling you and the trust and instinct on how to respond that can only come with practise.

There are no short cuts to get you to the end game, because no ‘killer product’ alone is going to solve all your problems. Your organisation can use killer tools, but it has to learn how to, hypothesis, test, iterate on what problems to solve with and hot to embed the tools in your processes to do so.

In this game of snakes and ladders, there are plenty of snakes and very few ladders. Only persistence, the right attitudes to test, fail, learn and relentless focus on shortening the feedback cycle of those loops will see you progressing faster across the data maturity spectrum.

About The Author

Matt Quinn is a Cloud Solution Architect at Microsoft UK. Part of the Customer Success team, Matt helps customers plan, architect and execute on their move to Microsoft Azure as part of their Digital Transformation Journey.