Multi-Cloud File Storage Design Considerations

How Efficient Metadata Handling is Essential to Maximizing the Value of File Data Across Multi-Cloud Infrastructure

The modern business climate has evolved dramatically with the rise of the digital transformation age. IT can no longer remain a subservient cost center. Instead, IT must leverage its expertise to transform into a critical business enabler of net new revenue and cost saving opportunities. According to ESG’s research on IT spending intentions, 86% of IT decision makers agree with the statement, “If we do not embrace digital transformation, we will be a less competitive and/or effective organization.”

For the business, the rewards are massive. For IT, however, the transformation is often daunting. IT must simultaneously find the extra budget and resources to architect and execute this end-to-end business transformation while doing its usual job of keeping the business running.

While multiple data types fuel these transformations, it is the file data environment that is often the most unwieldy and complex. This complexity stems from massive surges in both content generation and retention that have been constant for nearly a decade. The top three identified workloads driving data growth are digital media (31%), collaboration (30%), and business intelligence (29%), and all three comprise file content.

File growth, however, only scratches the surface of the real problem. For decades, the infrastructure designed to serve file content was architected for both preservation and cost control. With the emergence of analytics, artificial intelligence (AI), and the Internet of Things (IoT), however speed is now of the essence. Necessitating that the right data is identified, located, and isolated on high-performance infrastructure, quickly—or the business initiatives that leverage that data will suffer.

This is a small problem when you have a dozen or so terabytes (TBs) of data. Modern file environments, though, often measure 1,000 times larger, in the dozens of PBs or billions of files under management. Adding complexity, the rise of public cloud services and the edge has dispersed these massive data volumes across a disaggregated environment.

A glimmer of hope does exist, however, with recent innovations in metadata handling, and one innovator that you have likely not heard of, InfiniteIO, is taking a novel approach to addressing the challenge of multi-cloud file storage, one they define as the true hybrid cloud. The idea is simply that if metadata could be consolidated and then harnessed quickly and easily, then the file data universe would seemingly shrink and become far more manageable. To that end, InfiniteIO’s architecture abstracts and accelerates the metadata for all file content in a multi-cloud ecosystem with zero impact to applications, simultaneously accelerating multi-cloud file storage while minimizing the cost.

The Rise of Multi-Cloud File Storage and Disaggregated IT

Simultaneously scaling capacity and performance quickly becomes unsustainable with traditional NAS systems. In an ideal world, all data would live on the fastest storage tier, such as all-flash, but for massive scale, often multi-petabyte file environments, that is impractical and burdens IT.

In response, 35% of IT organizations have turned to the public cloud. As a result, businesses are opting to integrate multiple public cloud infrastructure providers into their data storage ecosystem. According to ESG research, 76% of IaaS users leverage more than one public cloud infrastructure provider.

Despite the benefits, the addition of cloud services introduces performance impacts and new cost considerations, often in the form of egress fees. Several of these new cost considerations are often underestimated by IT and the business. Additionally, regulatory compliance and data sensitivity often dictate where files can reside, further increasing both the need for a distributed infrastructure and the complexity of managing it. As a result, modern file storage environments are becoming increasingly disaggregated, complex, costly, and slow.

The Role of Metadata in Addressing Multi-Cloud File Storage Complexity

As hybrid and multi-cloud become the new normal, IT becomes increasingly disaggregated. The quantity, quality, and pace of data analysis and machine learning translate into great business opportunity but increasing the volume of data greatly diminishes the pace at which insights can be captured. The time lost is too expensive given the value of these initiatives. It is here that metadata handling can play an integral role.

For those not familiar with metadata, It is the data about the file data. In a typical file storage environment, most of the requests for files are actually for metadata, as file metadata includes attributes such as when was the file created, who created it, who can read it, and the file size. In addition, multiple applications, such as for backup and storage functions, actively scan metadata for changes.

When the metadata is stored on high-performing infrastructure, these activities take less time. Conversely, slow metadata responses drag down file storage usage as scans, directory views, and searches all take longer. These delays hinder operations on file data and slow down applications that require metadata scans—for example, increasing the time required to back up or replicate a file environment.

As file counts and capacities scale, systems that once offered high-performance access can slow significantly. For file environments that measure in the tens of petabytes and are distributed across multiple data centers, multiple clouds, and the edge, the right metadata storage architecture is essential.

Efficient metadata handling is required for not just one platform, but the entire multi-cloud file environment. As demands grow, efficient metadata handing will be essential for businesses to maximize the value of their file data, control storage infrastructure costs, and ensure the successful integration of public cloud infrastructure.

Transform File Storage with InfiniteIO’s True Hybrid Cloud

Vendors are often content to focus innovation solely on their own storage platform. As a result, few solutions address metadata handling at a multi-platform, multi-cloud, and multi-workload level. One emerging innovator, InfiniteIO, believes metadata is the key to unlocking the true hybrid cloud and has developed a novel approach to addressing this multi-cloud metadata issue.

InfiniteIO’s system, the NSC-110, delivers high-performance local metadata abstraction via deep packet inspection of both on- and off-premises file stores and puts the metadata in a memory-based metadata map. Leveraging memory to store metadata accelerates the metadata operations, and even massive, highly distributed file environments appear as if they reside fully on high-performance local storage.

By leveraging deep packet inspection to learn metadata in real time, the NSC-110 sits invisibly between the clients and the existing file systems, delivering the benefits of metadata acceleration without massive investments in new infrastructure, time-consuming and costly file migrations, or even disruptive changes to existing mount points. InfiniteIO then offers a variety of policy-based tools to automatically identify and then migrate data to the optimal tier in the multi-cloud file ecosystem, and all data movement is encrypted for better security.

This design stems from InfiniteIO’s true hybrid cloud ideal, which is built upon five fundamental principles:

The Transformation Benefits of Multi-Cloud Metadata Abstraction and Acceleration

Automatic location of inactive data across a heterogeneous multi-cloud environment

Automated tiering to archive or to the cloud without slowing down metadata

The Bigger Truth

File data is a strategic and incredibly valuable asset. Businesses that can quickly and effectively harness these assets find success, while those that cannot often struggle to stay relevant. This is the reality of the digital economy and the driving force behind digital transformation. Analytics, AI, and IoT are all predicated on the creation of content and the ability to quickly analyze the data, identify insights, and adjust decisions, processes, and strategies accordingly.

With continued growth in hybrid and multi-cloud adoption, the future of file storage will likely continue to be just as diverse as it is today, if not more so with capacity growth showing no signs of slowing down. InfiniteIO, however, makes diverse file ecosystems seem smaller, more manageable, and more accessible, without changing the existing environment. Business have greater freedom to choose the right platform for their data and more freedom to adjust. InfiniteIO calls this the true hybrid cloud. And with it, businesses are freed to focus attentions on leveraging their digital assets to their fullest extent, rather than spending their time just maintaining the status quo. While still a young player in the enterprise file storage space, InfiniteIO’s technology has tremendous potential. And its invisible, no-risk deployment approach makes it easier to experience what the technology can do.

Resources

Products

We use cookies to offer you a better browsing experience, analyze site traffic and serve targeted advertisements. Read about how we use cookies by reviewing our Terms of Use and Privacy Policy. If you continue to use this site, you consent to our use of cookies.