High Performance MPI Over PCIe

MPI Software for PCI Express network

Message Passing Interface (MPI) is a well defined programming API for High Performance Computing applications. It provides rich set of communication primitives for efficient communication between processes running on SMPs and clusters interconnected by various types of networks.

MPI over SISCI

The SISCI API provides an well established programming API for shared memory applications. NICEVT created a SISCI transport for MPICH2, this code is open source and maintained by engineers from Dolphin and A3Cube.

The software is currently available for Windows and Linux and based on MPICH2-1.0.2p1

Here --with-devcice=ch3:stream enables the shared memory transport for MPICH2 which NMPI has forked, and --with-sisci points to the top-level installation directory for the Dolphin software. /opt/DIS is the default. --prefix specifies the top-level directory for the NMPI installation.

The .mpd.conf file cannot be readable for other users, and needs to be present on each node (nfs-mounted home-directory or similar)

On the job-management node (which does not need to be a cluster node), you also need to have a machinefile, which specifies the nodes to run on (including the job-management -node if it is part of the cluster);

# cat machine
tiger-1
tiger-2
tiger-3
tiger-4

RUNNING MPI jobs

MPI applications are started with mpiexec, which will contact the MPDs running on the nodes to start the actual job. NMPI provides the mpdboot and mpdallexit -tools to control the MPDs. Mpdboot uses ssh to start the MPDs on the other nodes, and further job management and application output are sent over traditional sockets.

A typical session for running 2 MPI-jobs will be

mpdboot
mpiexec
mpiexec
mpdallexit

In detail:
# /opt/NMPI/bin/mpdboot -n 4 -f machine
(this will cause the current user session to ssh to each of the 4 first nodes in the './machine' file and spawn an MPD process. Path-information from the configure-line is compiled into mpdboot. )

# /opt/NMPI/bin/mpdtrace
tiger-1
tiger-2
tiger-3
tiger-4

(mpdtrace will list running nodes - optional for troubleshooting)

# /opt/NMPI/bin/mpiexec -n 2 /opt/NMPI/examples/cpi
(this will run the 'cpi' test-application that comes with NMPI, using 2 processors, 1 from each of the two first of the mpdbooted nodes)

# /opt/NMPI/bin/mpiexec -n 8 /opt/NMPI/examples/cpi
(Same as above, using 8 processors, assigned round-robin on the nodes. You do not need to run on all nodes booted, nor are you limited to only running one process pr node.)

# /opt/NMPI/bin/mpdallexit
(this pulls down the started MPDs, terminating the session).

NMPI for Windows

The header files, documentation and libraries are installed with the Development feature.

MPI over SuperSockets

The Dolphin SuperSockets or the TCPoPCIe software are also able to run all standard Linux MPI libraries that supports Ethernet. The performance highly depends on the Ethernet transport of the selected MPI library and will normally be significantly better than Ethernet, but not as fast as NMPI.