Cloud adoption Over the last year, we’ve seen a strong shift towards the use of cloud in HPC, particularly in the case of storage. Many research institutions

are working towards a ‘cloud- first’ policy, looking for cost savings in using the cloud rather than expanding their data centres with overheads, such as cooling, data and cluster management and certification requirements. There is a noticeable push

towards using HPC in the cloud and reducing the amount of compute infrastructure on-premise. Following the AMD agreement with cloud AWS and Azure and their respective implementation of technologies, such as Infiniband, into these HPC cloud scenarios, it’s becoming more likely to be the direction universities are heading in 2020. I don’t foresee cloud

completely replacing large local HPC clusters in the near future, but for customers with variable workloads, on-premise HPC clusters could become smaller and closely tie into the public cloud to allow for peaks in utilisation. Additionally, we’re increasingly seeing an uptake in using public cloud providers for ‘edge’ cases such as testing new technologies or spinning up environments for specific projects or events. With further understanding of the technologies involved and user requirements, most universities and research institutions are at least considering taking a hybrid approach.

Cloud storage One of the major downfalls of HPC in the cloud is the high cost of pulling the data back out of the cloud, which is understandably a cause for resistance in some organisations moving towards the cloud. However, there are products

coming onto the market from both NetApp and DDN that are ‘hybridised’ for the public cloud, whereby you are able to upload some of your storage into the public cloud, process it and only download the changed content.

This means only being

charged for the retrieval of the new data that is required. Only a year ago, every

storage vendor needed to have a cloud connector so organisations could move their data into the cloud and move it back in its entirety. The recognition by

these storage vendors that organisations don’t want to store all their data on the cloud and only move small amounts of data in and out, will avoid the huge expenditure of data retrieval and move the adoption of HPC in the cloud forward in 2020.

Containerisation and storage developments There is a big push on the parallel file system BeeGFS, now available on open-source, which is seeing some extremely positive bandwidth results within HPC compute clusters. There are storage vendors

who are now looking at containerising BeeGFS, so it can be included on embedded devices in the storage system to ensure faster deployment and configuration management.

file system in a virtualised environment is becoming increasingly popular, notably, IBM is looking at it for its IBM Spectrum Scale storage solution to ease the deployment of their IBM ESS product.

Containerisation allows you

to put your applications or file systems in a ‘wrapper’, so they become very mobile, with the ability to tie them into standard configuration management. By designing components of the cluster as a container in the lab, it allows for faster deployment, ease of management and upgrading on-premise. A lot of research institutions

are using containerisation to containerise their scientific applications and experiments, as it enables a researcher use of the entire HPC environment with all the libraries and applications for an experiment. The researcher can then

replicate the experiment multiple times around the cluster (emulating a 100-node job), running their experiment within this containerised environment, with very little dependencies on the host operating system or the administrator’s configuration of the cluster. Once the experiment is

complete, the researcher can archive the container which can then be easily reloaded multiple times on different occasions, making re-configuration much simpler and data retrieval more cost- effective.

Security The ability to restrict the containers and section up the memory, to avoid any memory

leaks, is certainly becoming more prominent in recent months.

Some providers are starting

to limit access to the same system, via a total encryption multi-tenant approach, which secures part of the memory between containers and virtual machines (VMs), so they aren’t able to see each other’s memory maps.

One of the major security aspects of cloud computing and containerisation is the concern that other users or tenants on the system are able to start looking at memory maps and leaking information of research that is confidential for example, medical research using non-anonymised data. Having new security

technologies coming onto the market whereby you are limiting the scope of the container or how the VM is able to access the memory goes a long way to reduce that worry.

GPUs There is no doubt whatsoever that GPU computing has become more significant with the rise in deep learning, used by artificial intelligence, data mining and data analytics. NVIDIA’s support for Arm-based HPC systems combined with its CUDA- accelerated computing is giving the HPC community a boost to exascale. ARM‘s ability to produce incredibly low powered CPUs has incredible benefits in an HPC environment. With many new technology

developments and positive uptake of cloud and containerisation, 2020 will herald exciting times for the HPC market.

Spring 2020 Scientific Computing World 13

containerisation to containerise their scientific applications and experiments, as it enables a