High Performance Computing Cluster in a Cloud Environment

High Performance Computing (HPC) enables scientists and researchers to
solve complex problems that require many computing capabilities. HPC
typically utilizes a message passing interface (MPI) to communicate
between different nodes.

HPC in Cloud

Currently, most projects requiring HPC are still running on legacy Unix
systems. Migrating these projects to a Cloud-based installation is very
simple and does not require much additional setup. In this tutorial, we
will build an HPC cluster with Open MPI on the Rackspace Cloud. Next, we
will run an Open MPI application on top of our cluster. By the end of
this tutorial, you will know how to leverage the Cloud to rapidly build
and scale an HPC cluster for real-time data processing while removing
the dependency on physical infrastructure.

Open MPI

To achieve high performance clustering in the Cloud, we will use Open
MPI, which is a Message Passing Interface project. It provides parallel
processing, thread safety and concurrency, dynamic process spawning, and
network and fault tolerance. This library is used by the world’s fastest
super computers and is instrumental in powering many petaflops. To find
out more about Open MPI library, visit their site:
http://www.open-mpi.org/.

Objective

In this tutorial, we will show you how to build an HPC cluster using the
following:

Four Rackspace Cloud Servers

Open MPI

Prerequisites

The following prerequisites are expected for successful completion of
this tutorial:

Installation Process

In this tutorial, we will be setting up a four-node cluster, running
applications on it, and gauging the performance.

Figure 1 - HPC on the Cloud High Level Architecture

In this tutorial, we will show you how to:

Create a Cloud Server

Install Open MPI

Enable Clustering

Configure HPC

Create and deploy a Cloud Server image

Install and run a sample Open MPI enabled application

Create a Cloud Server

Log in to the Cloud Control Panel and create a Cloud Server from
the web interface with the following attributes.

Server name: yourname-HPC-node-01

Region: Dallas (DFW)

Image (Rackspace): Ubuntu 12.04 LTS (Precise Pangolin)

Size: 2GB of RAM or higher

Click Create Server and the Cloud Server will begin building. During this time, a popup will
appear with the Cloud Server password. Record this information as you
will need it later. Dismiss the popup window and wait for the server
build to complete. Write down IP address for the server as it becomes
available.

Install Open MPI

Once the server finishes building and is in Available status, SSH into
it and log in using the IP address and password you recorded earlier.

Configure HPC

Now we are going to configure the master HPC node by creating a host
file. To do this, ensure you are logged in to the first node over SSH
and create the following file where is the IP
address you used to SSH into the machine:

cd ~/
echo <Your Server IP> >> mpi_hosts

To verify the file:

cat mpi_hosts

You should see:

<Your Server IP>

To verify we have configured everything correctly so far, we will use
the hello_c.c from the examples included with Open MPI.

Now that we have confirmed that the first node is online and
operational, we will finish building the cluster.

Create and Deploy a Cloud Server Image

With our first node created, we are ready to set up a cluster. We will
need to make copies of the node we just created. Log in to the Cloud Control Panel again
and follow these steps to create an image:

Navigate to the servers list

Select the server you created for the first node

Click on the Actions drop down menu

Click Create Image. See Figure 2 for details.

When prompted, provide a meaningful name as shown in Figure 3.

Finally, click Create Image and wait a few minutes for the image
to be created.

When completed, deploy a new Cloud Server using our prior procedure with
the following exception; when prompted for image, click the Saved tab.
Again, you will need to provide a meaningful name. Additionally, record
the password and IP address.

Let’s say that the IP of your new server is 10.20.30.40, and the
IP/hostname of your first server is . To add the
new node to the cluster, do the following:

SSH to your first server
cd ~/
cat >> mpi_hosts <<EOF
10.20.30.40
EOF

Now, your host file should resemble this:

<Your Server IP>
10.20.30.40

To test the connectivity between the nodes, execute the following
command:

mpirun -v -np 2 --hostfile ~/mpi_hosts /root/samples/connectivity

If you don’t get any errors, you have just successfully created and
tested your own tiny cloud cluster! To increase the size of the cluster,
add two more nodes using the same procedure.

To test the connectivity between the four-node cluster, execute the
following command:

mpirun -v -np 4 --hostfile ~/mpi_hosts /root/samples/connectivity

Install and Run a Sample Open MPI enabled Application

Now that we have an Open MPI cluster, let’s see how it performs. We will
use a simple ray tracing application that can run on a single node or on
an Open MPI cluster and compare the performance.

First, we will need to install this application on all nodes of the
cluster. To do this, SSH into the master node and do the following:

Our cluster consisted of four nodes and one CPU each, therefore, the
performance improvement was almost four times greater. Even if you don’t
have multiple nodes and run your application on one node, but tell
OpenMPI to use both CPUs, you will still have significant improvements.

Summary

In this tutorial, you learned how to create and image Cloud Servers. You
also learned how to setup an HPC cluster using Open MPI. After setting
up and configuring the cluster, you installed a small ray tracing
application to demonstrate the benefits of using multiple nodes instead
of one node.