Parallel Programming with MatlabMPI

Matlab is the dominant programming language for implementing numerical computations and is widely used for algorithm development, simulation, data reduction, testing and system evaluation. Many of these computations could benefit from faster execution on a parallel computer. There have been many previous attempts to provide an efficient mechanism for running Matlab programs on parallel computers. These efforts have faced numerous
challenges and none have received widespread acceptance.

In the world of parallel computing the Message Passing Interface (MPI)
is the de facto standard for implementing programs on multiple processors. MPI defines C and Fortran language functions for doing point-to-point communication in a parallel program. MPI has proven to be an effective model for implementing parallel programs and is used by many of the world's most demanding applications (weather modeling, weapons simulation, aircraft design, etc.).

MatlabMPI is set of Matlab scripts that implement a subset of MPI and allow any Matlab program to be run on a parallel computer. The key innovation of MatlabMPI is that it implements the widely used MPI "look and feel" on top of standard Matlab file i/o, resulting in a "pure" Matlab implementation that is exceedingly small (~300 lines of code). Thus, MatlabMPI will run on any combination of computers that Matlab supports. In addition, because of its small size, it is simple to download and use (and modify if you like).

On shared memory systems, MatlabMPI only requires a single Matlab license as each user is allowed to have many Matlab sessions. On distributed memory systems, MatlabMPI will require one Matlab license per machine. However, if your scripts are compilable with Matlab's mcc command then you only need one license no matter what.

Because MatlabMPI uses file I/O for communication, there must be a
directory that is visible to every machine (this is usually also required
in order to install Matlab). This directory defaults to the directory that
the program is launched from, but can be changed within the MatlabMPI program.

III. INSTALLING AND RUNNING:

1. Copy MatlabMPI into a location that is visible to all computers.

2. Add MatlabMPI/src directory to matlab path (e.g. put

addpath ~/MatlabMPI/src

in your matlab/startup.m file. You may require a different path than
the above statement, depending on where you install MatlabMPI).

MPI_Run has three arguments: the name of the MatlabMPI program (without the ".m" suffix), the number of machines to run the program on and the"machines" argument which contains a list of names of machines on which to run the program. If the user directs MPI_Run to use N machines and the "machines" list contains M names, where M > N, then MPI_Run will execute the program on the first N processors in the list.

The "machines" argument can be of the following form:

machines = {};
Run on a local processor.

machines = {'machine1' 'machine2'}) );
Run on multiprocessors.

machines = {'machine1:dir1' 'machine2:dir2'}) );
Run on multiprocessors and communicate via dir1 and dir2, which must be visible to both machines.

NOTE: The MPI_cc command will only work if you have properly configured your paths so that your code will compile (e.g. your LD_LIBRARY_PATH, etc ...), which can take a little effort. On Linux systems you will have to add (at a minimum):

matlab/extern/lib/glnx86
matlab/bin/glnx86
matlab/sys/os/glnx86

IV. LAUNCHING AND FILE I/O

Launching from the Matlab command line using eval(MPI_Run(...)) allows for a variety of behavior. MPI_Run(...) does two things. First, it
launches a Matlab script on the specified machines with output redirected to a file. Second, if the rank=0 process is being run on the local machine, MPI_Run(...) returns a string containing the commands to initialize MatlabMPI, which allows MatlabMPI to be invoked deep inside of any Matlab program.

This launching and output behavior is summarized below:

Machine

MPI Rank = 0

MPI Rank > 0

Local

exec() > screen

unix > file

Remote

rsh > file

rsh > file

The files that contain the program output are stored in the MatMPI
directory which is created by MPI_Run. Filenames are of the form:

program.rank.out

where "program" is the name of the MatlabMPI program and "rank" is the rank of the MatlabMPI process that generated the output. See ERROR HANDLING for more information about the MatMPI directory.

V. ERROR HANDLING

MatlabMPI handles errors pretty much the same as existing Matlab; however running hundreds of copies does bring up some additional issues.

– If an error is encountered and your Matlab script has an "exit" statement (which it should) then all the Matlab processes should die gracefully.

– If a Matlab job is waiting for a message that will never arrive, then
you will have to kill it by hand by starting a new Matlab session and
typing:

MPI_Abort

If this doesn't work, you will need to log into each machine, type
"top" and kill the Matlab processes one by one.

- MatlabMPI can leave a variety of files lying around, which are best
to delete once an error has occurred by starting Matlab and typing:

MatMPI_Delete_all

If this doesn't work, you can delete the files by hand. The files can
be found in two places - the launching directory and the communication directory (which by default are the same place).

In the launch directory you may have leftover files that look like:

MatMPI/*

In the communication directory you may have leftover files that look
like:

p<rank>_p<rank>_t<tag>_buffer.mat
p<rank>_p<rank>_t<tag>_lock.mat

It is very important that you delete these files. In general, if you
are using a public directory to do communication (e.g.; /tmp), you should create a subdirectory first (e.g.; /tmp/joe) and use this directory for communication.

VI. RUNNING ON LINUX

MatlabMPI now performs great on Linux clusters. The key is changing the"acdirmin" parameter from 30 seconds to 0.1 seconds when you NFS mount the file system that is used for communication. Here is the mount command we found that worked well for us:

1. Put a copy of Matlab on each node and alias the 'matlab'
command to the copy on each of the nodes.

2. Cross-mount all the local disks so each node is receiving
files on its own disk.

NOTE: With these optimizations, sometimes Matlab can launch too fast with weird NFS errors as a result. We are not sure of the cause, but we have found that removing all path statements from your .cshrc file can help.

– Sometimes odd behavior results when very large (~100 MByte) messages are used. If you notice problems, you might try sending several smaller messages.

– Sometimes a large amount of NFS GETATTR traffic can be generated if you are waiting a long time to receive files. You can probe this traffic by logging into the NFS server as root and typing
"snoop | grep <machine-name>", and watch for "GETATTR" packets.

You can either edit this file directly or copy it into the directory you
are running out of and edit it locally.

By default, MatlabMPI uses rsh. Unfortunately, switching from rsh to ssh requires more than just changing the machine_db_settings.remote_launch setting. The following procedure will allow you to use ssh without being prompted for a password (a necessary requirement for MatlabMPI):

Type "which matlab" on each of the machines you are trying to run on.
If it says:

matlab: Command not found

then you need to put it in your path. You can also hard code the path
into MatlabMPI by editing the MatlabMPI/src/MatMPI_Comm_settings.m and changing the line:

matlab_location = ' matlab ';

Note that you still must be able to run Matlab from the shell prompt on the machine on which you are launching your MatlabMPI program.

3. Test that MatlabMPI is in your path.

Start matlab and type:

help MatlabMPI

if you get the error:

MatlabMPI not found

then you need to put MatlabMPI/src in you path. See the INSTALLING
AND RUNNING instructions in README and README.pc.

4. Starting up (xbasic script).

Go to the MatlabMPI/src/examples directory. Edit RUN.m so that it
only runs the 'xbasic' script. Also, change the machines variable so
that MatlabMPI runs on the machines that you want. Start matlab and
type RUN. This should run a few seconds and return 'SUCCESS'. Quit
matlab and look at the output of the other process, for example type:

more MatMPI/*.out

If it ran successfully, great. If not, take a look at the error
message.

5. Testing MatlabMPI

There are man scripts in the examples directory which are useful
for testing performance. In this section we explaing how they can
be used.

a. Start up peformance (basic script)

Repeat step 4 with the 'basic' script. 'basic' sends a message
between two processors. If it hangs, kill matlab and look at the
output. If it runs successfully, run it a few times and note the
runtimes. The first time may take 20 seconds, subsequent times
should take ~2 seconds. If it takes a lot longer, Matlab may be
taking a long time to launch on remote machines. There are two
likely causes.

The first is that sometimes your $PATH variable can take a long time
to evaluate. Comment out various environment variables and see if it
affects performance.

The second is the latency in launching Matlab. This latency can
sometimes be alleviated by installing a local copy of Matlab on each
machine so it doesn't have to be sent over your network.

b. Messaging peformance (multi_basic script)

Repeat step 4 with the 'multi_basic'. 'multi_basic' sends several
messages between two processors. It is very useful for debugging
performance problems that can occur when you cross-mount file systems to achieve better messaging performance. If multi_basic runs successfully, run it a few times and note the runtimes. The first time may take 20 seconds, subsequent times should take ~2 seconds. If it takes a lot longer and you notice lots of times around integer seconds (e.g. 1 second, 2 seconds, 30 seconds), then this most likely means that there are some lags in updating the cache on your filesystem. This is particularly common in Linux. See Section VI - RUNNING ON LINUX for good ways to mount the file system.

c. High peformance messaging (speedtest script)

Repeat step 4 with the 'speedtest' script. 'speedtest' sends many
messages of increasing size between two (or more) processors. It is
very useful for testing the limits of your filesystem. It might take
several minutes to run. After it has run, type:

total_time

You will see a listing of the times it took to send all the messages.
These should start with small numbers (~50 millesconds) and eventually get up to around 1 second or more. Next type:

bandwidth

This is the simultaneous bandwidth between the two processors. It
should start out small, and then grow to around 10 Megabytes/ second or greater. You should also look at the results of speedtest from other processors which are saved in speedtest.1.mat, ...

d. High Performance computation (blurimage script)

Repeat the procedure in Step 4 with the 'blurimage' script. This
filters an image and reports the performance in Gigaflops.
It is a good test of performance.

e. Testing everything (unit_test_all)

Edit the unit_test_all script to run on the machines you want (be
sure and remember the "&" for the first machine). This script will
test everything. It can take a while. There is also a similar script
called unit_test_all_mcc, which tests all the scripts with the compiler
command (MPI_cc).

6. Testing your own program.

Hopefully the above scripts will give you some insight into how to make your own program run better. Here are some more specific instructions.

a. Timing

Time the part of the program you want to run in parallel. Rememember that MatlabMPI only can improve this part of the program. Furthermore, as the parallel part runs faster, the non-parallel part will start dominating the overall time (a.k.a. Amdahl's law).

Note that while parallization can reduce computation time, it adds
overhead due to sending messages. Compare the times you get for your messages with times in speedtest. You should also time how long it takes to send messages in your program. If your program spends too much time performing communication, your program may not benefit from the reduced time in computation. Which leads us to the next section...

b. General Tuning

Minimize communication. Try and do as much computation as you can
without sending messages. Also, it is much much better to send a few big messages than many little messages.

You should try and divide the work among the different processors so that it is even as possible (this is called Load Balancing). If the
workload varies as a function of the data, than you might want to set
up a client/server model that deals out work on an as needed basis.

Avoid restart. You can start MatlabMPI anywhere in your program,
however you should also avoid restarting MatlabMPI in your program.

XIII. FIRST-TIME USER'S RULES OF THUMB

Thanks to Theresa Meuse for starting this section.

– Delete stale copies of the MatMPI directory prior to launching your program. See the next item for more details.

– Use the RUN.m script included in the examples directory to run your program. RUN.m kills stale Matlab processes leftover from a prior MatlabMPI program, removes the MatMPI directory, then launches your program.

– Make sure the directory containing a MatlabMPI program is not write-
protected; MatlabMPI needs to create and write to the MatMPI directory. In Windows, we have seen directories become write-protected when stepping through a MatlabMPI program with the Matlab debugger without completing execution.

– Make sure that when installing a new version of MatlabMPI that all source code from the old version is deleted or that your path points to only the new version. Make sure that there are no copies of any old MatlabMPI source code in your current working directory. This leads us to the next point:

– Don't change the directory structure of MatlabMPI. To quote the old adage:"If it ain't broke, don't fix it." Changing the directory structure may inadvertently lead your application to behave differently than expected.

– Make sure your application does not use "clear" or "clear all" at any point within your MatlabMPI code. When you run:

eval( MPI_Run('...', ..., ...) );

a global variable called MPI_COMM_WORLD is created. Almost all the MPI functions require the information contained in MPI_COMM_WORLD.

– Make sure that your application does not change its working directory during execution. Otherwise, MatlabMPI cannot find/access the MatMPI directory.

– Tag management is crucial. Take care when creating and managing tag values.

– READ SECTION XII - DIAGNOSTICS AND TROUBLESHOOTING. Start with the examples and be able to execute them on your local machine only. Then get the examples working on multiple machines. Then develop an example for yourself.

– Exiting and restarting Matlab between runs of MatlabMPI programs can improve performance. Restarting Matlab performs a number of tasks, e.g. clearing out memory, closing file, etc., that can potentially improve performance of MatlabMPI programs running in a new instance of Matlab.

THIS SOFTWARE IS PROVIDED "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

MATLAB® is a registered MathWorks trademark. Reference to commercial products, tradenames, trademarks, or manufacturer does not constitute or imply endorsement.