MPI

Overview

MPI is an acronym for Message Passing Interface, which is a standard Application Programming Interface (API) for parallel programming in C, C++ and Fortran on distributed memory machines.

MPI is implemented as a library of functions, a C, C++ or Fortran programmer doesn’t have to learn a new language to use MPI; if a sequential program already exists, much of that program can be used for an MPI version. An MPI program complies just like a regular program. At link time, the MPI library must be accessed.

On clusters, each node has a separate memory space, i.e. distributed memory, as a result, there is no way to get at another’s except via message passing over the network. Messages may be used for:

Sending data

Performing operations on data

Synchronization between tasks

For efficiency, MPI is usually run on clusters of many computers of the same type, with high-speed connection. MPI could also be run (with less reliable speedup) on collections of several types of computers with relative slow connections. Due to distributed memory between processors, the data for the problem has been divided up among these processors. This means that, from time to time, one processor may need to “ask” another processor for a copy of certain data or result.

Another attractive approach to parallel programming is called OpenMP. This system is used when a shared memorymachine is available, this is when several processors share a single physical memory, or a number of processors can address the same “logical” memory. An existing sequential program can be turned into an OpenMP program simply by inserting special comment statements.

MPI Code Structure

MPI has a wide range of capabilities. There are three subsets of functionality:

Basic (about 6 functions)

Intermediate

Advanced (up to 125 functions)

All processes must initialize and finalize MPI

MPI_Init(): Starts up the MPI runtime environment

MPI_Finalize(): Shuts down the MPI runtime environment

MPI uses MPI_Comm objects to define subsets of processors (i.e. communicators) that may communicate with one another. By default MPI_COMM_WORLD includes all your processors.

MPI Execution

Single Program, Multiple Data (SPMD) means every process gets a copy of the executable. Each process looks at its own rank (unique ID within a communicator) to determine which part of the problem to work on. Except when communicating, each process works completely independently.

Point-to-Point (P2P) Communication

Point-to-Point (P2P) Communication sends data from one point to another, one task sends while another receives.

Basic P2P involves two functions, MPI_Send() and MPI_MPI_Recv(). An alternative is MPI_SendRecv with different tags to indicate send and receive stages. It’s especially useful when each node both sends and receives messages (two-way communication).

Synchronous Communication: Messages are sent and received via MPI_Ssend() and MPI_Srecv(). Handshaking occurs between send and receive tasks to confirm a safe send. As a result, in some case, it blocks send/receive.

Buffered Communication: Messages are sent and received via MPI_Bsend() and MPI_Brecv(). The content of the message is copied into a system-controlled block of memory (system buffer). The sender continues executing other tasks; when the receiver is ready to receive, the system simply copies the buffered message into the appropriate memory location.

Blocking vs. Non-blocking

MPI communications can be blocking or non-blocking:

A Blocking (MPI_Send() and MPI_MPI_Recv()) send routing will only return after it is safe to modify the buffer. “Safe” in this case means that modification will not affect the data to be sent. Safe does not imply that the data was actually received.

Non-blocking (MPI_Isend() and MPI_Irecv()) send/receive routines return immediately. Non-blocking operations request that the MPI library perform the operation “when possible”. It is unsafe to modify the buffer until the requested operation has been performed. Non-blocking communications are primarily used to overlap computation with communication, optimizing performance.

Deadlock

Improper use of blocking receive/send will result in deadlock, where two processors cannot progress because each of them is waiting on the other. (This can also happen for many processors.) For example, the following code contains a deadlock: