Dark data: A complex challenge for enterprises and CIOs in the cloud era

Bangalore: Organisations and enterprises are dealing with IT in mainly two aspects - modernising the IT infrastructure or/ and digital transformation of businesses. In both cases, an enormous amount of data gets generated of which some is structured and the rest, at large is unstructured.

However, this newly generated data is in addition to the already existing huge old data volumes in the organisation.

Data is considered as the new oil for businesses today; however, enterprises and their CIOs face a big dilemma, whether to store or delete that old data. Ultimately, today’s enterprises rely more on data assets than ever in the past to get business insights and build strategies. And, that is where the big dilemma turns into a complex and costly affair in most organisations.

Dark dataWith no clear policy around data management in terms of the duration of stored data, its usage, security as well as deletion, most organisations end-up having, what’s called the ‘dark data.’

Dark data, in simple terms, is actually the unused data or information assets hoarding in the organisation. Many industry surveys have highlighted the impact of dark data on enterprises in recent years.

Over 50% of enterprise data is dark data, according to information management company Veritas. As much as 33% of enterprise data goes untouched and unleveraged, as per Data Genomics Index 2017.

And the problem of data storage doesn’t end here as data is expected to grow multi-folds in the near future.

According to the Cambrian Explosion of Data, the data growth was 6 ZB in 2014 and is expected to be 52 ZB in 2020. And the key drivers behind this unprecedented data growth are unstructured data, big data and web-scale workloads, and IoT.

Investment in data storageOverall, the increasing data and its management compel enterprises to invest heavily in IT infrastructure, mainly the storage infrastructure, whether it is on-premise, in the datacentre or in the cloud. And dark data is one of the main reasons why enterprises continue to invest in storage.

“If you ask the question -- are folks using and deleting the data which is not useful now? The answer is still no and I don’t see most customers are on that path yet,” says Balaji Rao, Managing Director –India & SAARC, Veritas.

Given the increasing storage cost and limited IT budgets, Rao points out that enterprises and CIOs are faced with crucial questions such as whether to store data for a long or short-term or should they use cloud storage for long-term data as it is cheap against on-premise and so on.

Multi-cloud environmentsHowever, this scenario becomes more complex because most organisations are leveraging multiple datacentres and clouds that make data and application management extremely challenging for their CIOs.

“Yes, it is adding huge complexities,” agrees Rao. He explains that earlier managing one datacentre itself was not so easy thing and now they have to manage multiples.

“Of course, the cloud service providers do offer some good CoEs (centre of excellence) and tools to simplify things. But the fact is organisations are using 2 or 3 different cloud environments like AWS, Azure, Google or hybrid cloud has become today’s reality,” states Rao.

This trend is highly seen among medium-size organisations that not so regulated. Such organisations face the complexity around their ability to move data and being resilient and provide a high level of availability to the customer and at the same time backup the data securely.

Cloud adoptionIn 2016, there were about 300 hyper-scale datacentres and is predicted to touch 600 by 2021. “And we do see trends where 94% of workloads are going to run in the cloud by 2021 – private or public and only 6% will be traditional datacentre style,” comments Rao on the rising cloud trend among organisations.

However, Rao informs that BFSI and Telecom sectors are still highly regulated from the cloud standpoint, but that’s not the case in other sectors, particularly in the mid-market and below where cloud adoption is higher than usual. “And of course, ITeS companies, which are a lot more on the technology front and less regulated,” he adds.

So data management and security-related complexities continue to remain for organisations and their CIOs whether it is dark data or in the cloud. While this is a global phenomenon, the situation is no different for Indian enterprises and CIOs here.

Data visibility“Yes, I am pretty sure and most of them have voiced out some of these concerns,” says Rao. Since organisations in India lack data visibility they don’t delete data at the time of business continuity (BC), they end up restoring all the junk data, which takes a long time.

“But, if I have data visibility, where I have only the usable data and keep it clean, then even my data recovery and resiliency would be a lot easier and my advent to the cloud would also be easier,” explains Rao. Further, he says, it would cost less on storage in the cloud because only useful data is moved to the cloud and archiving all the non-useful data.

“Even in the cloud, you have cheaper and expensive storages based on SLAs and throughputs, so I could choose the cheapest form of storage if I know this data is not something I need regularly,” he comments.

The data visibility will help organisations and CIOs to improve efficiency with business continuity (BC) along with overall compliance regulations including data privacy mainly because of data identification.

Rao believes that CIOs and organisations in India have realised the data visibility aspect when it comes to dark data and they are also taking some steps in this direction but are not yet there. “They are quite some distance away from it,” concludes Rao.