Big Data, Private Cloud and lessons from Google

The move to Big Data collection and processing methods means that organizations need ever larger data stores to hold information, and Cloud technologies are evolving to meet that need, by offering on-demand scalability so that there are always resources available.

The opportunity to outsource responsibility for building the necessary data storage infrastructure is attractive, and there are dozens of providers happy to oblige. But although there may be headline cost savings associated with the move to hosted Cloud storage, expanding capacity needs will result in larger subscription fees over time.

Building private Cloud onsite first

Clearly instant scalability is the way forward, but outsourcing may not be necessary. Taking a leaf out of Google’s playbook, your business could put existing storage hardware assets to use, providing the infrastructure for Cloud clusters.

Ever since its inception, Google has used ultra low-cost hardware to build search and indexing clusters, helping to maximize profit margins in the process and provide the computing storage and power needed for their operations. Repurposing legacy and properly supported post-warranty hardware allows businesses to use a similar cost-effective approach to building their own private Cloud, which they then retain complete control over – just like Google.

Big Data and the importance of on-site storage

With the exception of archive data, high speed access and processing is vital. Big Data systems that rely on ‘bursting’ to the Cloud, or that are entirely Cloud-based, are by their very design slower than a system based entirely on site. As businesses begin to use Big Data to automate processes, latency will become an even bigger issue, giving those who choose an on-site system a distinct advantage over their off-site competitors.

Repurposing existing hardware makes perfect sense in a Big Data scenario – the investment has already been made, the hardware is installed and ready to use, and upgrades of post-warranty hardware are far cheaper than purchasing all-new systems. So in many cases, looking backwards (as Google did for many years), could be the way forward.