High performance computing and the hybrid cloud

John Kirkley has been an editor and writer in the high tech world for more than 40 years, including serving as editor of Datamation magazine for over a decade. He created Computer magazine for the IEEE Computer Society and was its first editor. He was also the senior editor of The Exascale Report and the founding editor of the Digital Manufacturing Report.

Hybrid computing provides a unified interface to HPC resources located around the world, including all the major HPC cloud providers and supercomputer centers

December 10, 2018

High performance computing in the cloud is beginning to make inroads into the enterprise. The stage is being set for the widespread adoption of hybrid HPC systems that combine on-premise services with public cloud capabilities.

But the reality is that this is not going to happen overnight.

The cloud supercomputer

“Despite the benefits offered by HPC in the cloud, the majority of HPC work is still being handled by on-premise systems,” says Gabriel Broner, vice president and general manager of Rescale.

But as these users have discovered, there are some significant downsides to owning and operating an on-premise system.

For example, HPC systems often come with an extremely high price tag. However, IT is under constant pressure to reduce costs while doing more with less. Requests for major capital expenditures in the form of an HPC cluster may meet with resistance – especially as it is extremely difficult to predict the engineering needs that will impact the HPC cluster over the life of the system. Rapid technology changes could obsolete the cluster before payback is achieved.

On-premise HPC clusters are often designed to meet typical estimated workflows and cannot smoothly handle unexpected peak demand. In addition procuring and installing a new HPC cluster can take at least six months, meaning that IT is working with systems that are already considered outdated by the time they are installed.

One viable solution is to adopt cloud computing. Because the cloud is fully enabled and incorporates resources from different infrastructure providers around the world, companies are able to have access to and pay for only the capabilities needed for their specific workflows.

Cloud HPC providers offer full support for the simulation process on one platform including such essentials as software, mathematical tools, optimizers, meshing software visualization tools and security. Engineers and scientists are able to work with resources optimized for their workloads.

“Our customers are already doing amazing work,” says Broner. “For example, Boom Supersonic is designing Mach 2.2 supersonic airliners that will cost far less than the Concorde - a seat on the Boom airliner will cost what a business class seat on a conventional airliner costs today. The advanced jets have been designed totally in the cloud using computational fluid dynamics (CFD) and other modeling and simulation software specifically designed to run on HPC systems that are part of Rescale’s global platform.”

A manufacturer in California is a typical customer with an atypical product - space rockets. Its HPC utilization runs in peaks and valleys with the peaks occurring as the company gears up for a launch. On conventional HPC systems, running all the simulations in preparation for a mission took up to three months. By moving the simulations to the cloud, the company cut the time to days or in some cases, hours.

In another recent procurement, major Japanese car manufacturer Nissan was considering purchasing a large on-premise HPC system until it compared the scalability and the cost of owning and operating the big cluster with the pay as you go approach being offered in the cloud. Time was also a factor – it would have taken six months or more for the investment in the new machine to begin paying off from a total cost of ownership (TCO) perspective. Instead, the manufacturer was able to begin running simulations of the next generation of cars within three months after moving its design operations into the cloud.

This particular deployment is an example of hybrid computing – a new paradigm that combines a company’s on-premise HPC capabilities with remote HPC in the cloud operating through an easy-to-use unified interface. This hybrid system creates a single environment for both IT and engineering teams. IT managers have full visibility into all resources. Engineers are able to execute jobs between internal and external hardware as if they were addressing a single system. Because workloads can move between on premise, private cloud and third party public cloud services as computing needs and costs change, the enterprise gains greater flexibility and more data deployment options.

This approach provides HPC users with immediate access to the best architecture for each specific job without having to wait for the procurement of appropriate hardware and software. Delays caused by overloaded queues are avoided. The hybrid system also allows users to match the peaks and valleys of their workloads by having the requisite HPC hardware and software immediately available on a pay per use basis.

“Hybrid computing is helping IT managers and other HPC technologists rethink the deployment and use of the cloud for maximum efficiency and effectiveness,” according to Broner. “Instead of having one system sitting on premises, you now have access to multiple cloud providers and supercomputer centers around the world.”

He adds that an administrative portal allows users to create and administer a variety of HPC-related tasks, such as setting up users, groups, or managing resources to realize the best price/performance on the system. The portal hides the complexity of HPC allowing users who are not HPC experts to access all the capabilities available through the hybrid platform. They can write and run scripts – for example, for a job designing an advanced airfoil, the user clicks on a CFD application, adds input files, gets an immediate cost estimate, and hits run. This simplified approach allows new users to access HPC without having to learn the complexities of directly interfacing with the on-premise system and multiple cloud providers.

The hybrid approach will bring the benefits of HPC to a whole new class of users – small to medium-sized manufacturers (SMMs). There are about 300,000 manufacturers in the United States and over 95 percent of them can be characterized as SMMs. These companies, usually with fewer than 2,000 employees, have traditionally been reluctant to adopt HPC primarily because of the cost and the need to hire new expert personnel to run the system. Until recently, getting by by running simulations on an aging workstation has been the solution of choice.

But now, claims Broner, hybrid computing with pay-as-you-go pricing, along with simplified, unified interfaces to all the resources available in the cloud, has made HPC understandable and affordable for SMMs. He likes to cite the example of an antenna manufacturer whose products must be built or ordered to match the frequency of the intended application. This meant building large, heavy prototypes for testing, a process that often took months on a workstation

By using the HPC simulation software available through the cloud, the engineers were able to design and test a customized antenna in hours, coming up with a solution that was low cost and lightweight – the antenna fits in the palm of your hand. The use of hybrid computing has also allowed this SMM to explore new markets that were inaccessible before.

Among the primary benefits to users of today’s hybrid computing solutions, Broner cites immediate access to the best architecture for your specific workload.

Hybrid computing provides a unified interface to HPC resources located around the world, including all the major HPC cloud providers and supercomputer centers. There is no queuing up to get your job run or waiting for the installation of additional hardware and software. Based on your workload’s parameters, the system chooses the best available resources to run the individual job in the least amount of time and at the lowest possible cost.

This approach allows small companies to realize the benefits of HPC for the first time, while large organizations can ride the peaks and valleys of utilization in the most advantageous way. It appears that hybrid computing has a bright future.

This article was produced as part of Intel’s editorial articles program