Introduction

Running serial R code on the cluster is similar to running any other serial
job. We give a serial example, and then demonstrate how to run parallel jobs
using the snow and Rmpi packages. Make sure you've read the
tutorial for C programs
first, to understand the basics of serial and parallel programming on maya.

For more information about the software, see the
R
website.
To use R, load the R module.

[araim1@maya-usr1 ~]$ module load R/3.1.2
[araim1@maya-usr1 ~]$

You can check the version using the command below.

[araim1@maya-usr1 ~]$ R --version
R version 3.1.2 (2014-10-31) -- "Pumpkin Helmet"
Copyright (C) 2014 The R Foundation for Statistical Computing
Platform: x86_64-unknown-linux-gnu (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under the terms of the
GNU General Public License versions 2 or 3.
For more information about these matters see
http://www.gnu.org/licenses/.
[araim1@maya-usr1 ~]$

Serial example

Let us demonstrate running a simple serial example on the compute nodes.
Consider the following R script.

Here is a batch script to submit the job. We use the Rscript command to
execute the script. There are other possible methods such as
"R CMD BATCH driver.R", whose behavior may vary (e.g. echoing your
commands to the output and automatically saving your workspace before
exiting).

Serial example with plot

On maya, some special steps are needed to produce some types of graphics such
as PNG and JPEG. (PDF graphics do not require these steps). We will demonstrate
with the following script which creates a scatter plot and adds a regression
line.

The messages in slurm.err are status messages from Xvfb. Notice that we have
the graphic mtcars.png in the output. If mtcars.png is zero bytes, it likely
means that the virtual display did not work correctly. Viewing the image
should produce a result as follows.

Preparing to use parallel libraries

We have made several parallel R packages available for all cluster users.
They have all been prepared with OpenMPI 1.4.3 + GCC (module
"openmpi/gcc/64"). You may install your own versions of the libraries
locally as well, but for the rest of this page we will assume you are using
ours. To prepare your environment for the cluster-wide parallel R libraries,
ensure that the following modules are loaded.

Your .libPaths() may not be exactly the same, but should have similar entries.
The first entry contains cluster-wide libraries, and the second
entry contains the user's locally installed libraries. The local library
directory may not exist for you if you haven't installed any libraries for
yourself yet.

A snow example

Now we'll look at running parallel R jobs with snow. snow (Simple Network Of
Workstations) provides a simple but powerful programming model for solving
"embarassingly parallel" problems with R. For more information about
programming snow, see
snow Simplified
or Luke Tierney's
web site.

Two versions of simple snow scripts will be presented: one using
socket-based communication, and one using MPI communication. The programmer
interacts with the same high-level snow interface using either communication
method. However, the code to initialize the snow cluster, as well as the
performance, differs between methods. In our current setup, socket-based
seems to outperform MPI significantly, so that is the recommended method.
However, note that socket-based clusters are limited to a maximum of 126
workers.

snow with socket communication

Note that the snowslurm package is not yet available on maya

snow with sockets is implemented specially on maya because most users don't
have access to SSH to compute nodes. Instead of SSH, srun is used to launch
snow workers. This has been implemented in a simple R package called
snowslurm, which is already
set up on the cluster. We'll demonstrate by running the following code.

We set up a snow cluster from our assigned nodes, print out the hostnames of
the nodes, and perform a few simple parallel calculations before stopping the
snow cluster and exiting. Notice that we use the "snowslurm" library rather
than "snow" on the first line. This is our customized version of snow with
sockets
for maya.

When using "makeSLURMcluster" to create a cluster, we expect to be running
in batch mode from a SLURM submission script. If we try to run this script
from the R prompt like a normal R script, it will fail. Here is a sample batch
script.

Notice that we first load the snow library, then we find our number of
assigned MPI processes from an MPI call (using the Rmpi interface, introduced
briefly in the next section). We set up a snow cluster from our assigned nodes,
print out some information from the cluster members, and perform a few simple
parallel calculations before stopping the snow cluster and exiting.

In order to run our code on the cluster, we'll need a batch script as usual

Notice that we're requesting six processes in the develop queue, and launching
R as a batch process. There is one notable difference from usual SLURM scripts
here - we have only requested mpirun to launch a single process. That process will
be responsible for asking MPI to start the remaining processes that we had requested.
Now let's submit this script to the batch system.

Rmpi examples: master/slave mode

Rmpi is another popular R package for parallel computing. It's a bit more
involved to use than snow, but also offers more control. The usual use
of Rmpi follows a slightly different paradigm than traditional MPI.
In Rmpi there is a master process that spawns slaves to work
in parallel; the master usually maintains control of the overall execution.
In contrast, the traditional MPI paradigm is "single program multiple data"
(SPMD) where all processes are treated as equal peers, but processes may take
on specific roles during the course of a program.

In the current section, we show how to run Rmpi in the master/slave mode.
In the next section, we switch to the single program
multiple data paradigm.
We have noted that the performance of master/slave mode on maya is
significantly worse than SPMD mode, but the reason is not yet apparent.

Rmpi features many familiar communications like "bcast" and "send".
For more information about using Rmpi, a good place to check is the reference
manual on
CRAN.
Also see this
tutorial.

Hello example

To see Rmpi in action, we will run the following parallel "hello world" R script

Note that if the option the option "dellog = FALSE" is not specified, a
warning will be returned from OpenMPI when the Rmpi tries to clean up:

An MPI process has executed an operation involving a call to the
"fork()" system call to create a child process. Open MPI is currently
operating in a condition that could result in memory corruption or
other system errors...

The option "needlog = FALSE" tells Rmpi not to create the extra log files
associated with this cleanup. If you would like to see the log files
along with your results, do not specify this option.

To run this code, we will use the following SLURM script, which launches only the
master process initially, as in the snow example from earlier. When the master
calls "mpi.spawn.Rslaves", the remaining MPI processes will be initialized.

Gather & reduce example

Let's look at a slightly more interesting example of Rmpi, using the
"gather" and "reduce" communications. Notice especially that the master
process must be involved in the communication, which is invoked separately
from the slaves. (If you forget to include the master, the program will
hang until it is killed). Here is the R code

Rmpi examples: SPMD mode

Now we present examples similar to the previous section, except in an SPMD
mode. Notice that commands here are issued from the perspective of the
current process, and some of the master/slave commands may not make sense
here.

[araim1@maya-usr1 Rmpi-hello-spmd]$ sbatch run.slurm
[araim1@maya-usr1 Rmpi-hello-spmd]$ cat slurm.err
[araim1@maya-usr1 Rmpi-hello-spmd]$ cat slurm.out
Hello world from process 001 of 016, on host n1
Hello world from process 003 of 016, on host n1
Hello world from process 004 of 016, on host n1
Hello world from process 010 of 016, on host n2
Hello world from process 011 of 016, on host n2
Hello world from process 013 of 016, on host n2
Hello world from process 009 of 016, on host n2
Hello world from process 002 of 016, on host n1
Hello world from process 008 of 016, on host n2
Hello world from process 012 of 016, on host n2
Hello world from process 014 of 016, on host n2
Hello world from process 000 of 016, on host n1
Hello world from process 015 of 016, on host n2
Hello world from process 007 of 016, on host n1
[araim1@maya-usr1 Rmpi-hello-spmd]$

Demos in package 'pbdMPI':
allgather pbdMPI An example of allgather.
allreduce pbdMPI An example of allreduce.
any_all pbdMPI An example of any and all.
bcast pbdMPI An example of bcast.
divide pbdMPI An example of dividing jobs.
gather pbdMPI An example of gather.
pbdApply pbdMPI An example of parallel apply.
pbdLapply pbdMPI An example of parallel lapply.
reduce pbdMPI An example of reduce.
scatter pbdMPI An example of scatter.
seed pbdMPI An example of random number generation.
sort pbdMPI An example of sorting.