Environment

My configuration is as follow:

- my Management Node (MN) does not have InfiniBand but my Compute Nodes (CNs) do, which means OpenMPI will have to be recompiled on a compute node so the configure script will detect the IB libraries and compile OpenMPI with support for the IB fabric. Note that we could also install the IB stack on the MN but as the MN is the most critical piece of the cluster I would just install what is essential on it.

We need first to get the environment ready and download OpenMPI (both the stable and the feature version)

Where modules will look for module files is controlled by the following environment variable : MODULEPATH

So for example, if I want to create specific environment modules for the user root only I can put them in the following directory : /root/modules, add the following statement to the .bashrc :

export MODULEPATH=$MODULEPATH:$HOME/modules

What should be our strategy here ?
We want:

1- to be able to switch between the OpenMPI stable and feature releases
2- the framework to prevent us from loading MPI environments one on top of the other
3- root to access the environment but also make the environments available for all users

For point 1, we just need 2 environment files : easy

For point 2, we need to introduce conflicts in the module file. As basically all the MPIs will conflict with each other, the best is to put all the MPI modules we create in a directory called MPI for example, and use this directory for the conflict directive in the module file. Note that as long as we don't modify the module file for IBM Platform MPI, we won't be able to prevent any user from loading the PMPI env on top of any OpenMPI environment, and of course the latest loaded will take precedence as all of them are using the prepend statement for the environment setup.

For point 3, we can have /root/modules/MPI for root, but this is local to one node so it is anyway better to make the files available for everyone. If we look carefuly at the output of module avail, we can see all the files are under /usr/share/Modules/modulefiles and rpm -qf tells us this directory is created/owned by the modules rpm:

Note the last statement for LD_LIBRARY_PATH is not absolutely needed, mpirun will take care of that. I included it for one specific case: when you recompile hp-linpack and want to do a quick try locally (i.e. not using mpirun) the binary needs to know where to find the libraries.

Job <module avail> was submitted from host <411head> by user <testuser1> in cluster <phpc_cluster>.
Job was executed on host(s) <compute003>, in queue <medium_priority>, as user <testuser1> in cluster <phpc_cluster>.
</home/testuser1> was used as the home directory.
</home/testuser1/tempo> was used as the working directory.
Started at Wed Oct 9 15:40:16 2013
Results reported at Wed Oct 9 15:40:16 2013

An explanation of the input/output parameters follows:
T/V : Wall time / encoded variant.
N : The order of the coefficient matrix A.
NB : The partitioning blocking factor.
P : The number of process rows.
Q : The number of process columns.
Time : Time in seconds to solve the linear system.
Gflops : Rate of execution for solving the linear system.

An explanation of the input/output parameters follows:
T/V : Wall time / encoded variant.
N : The order of the coefficient matrix A.
NB : The partitioning blocking factor.
P : The number of process rows.
Q : The number of process columns.
Time : Time in seconds to solve the linear system.
Gflops : Rate of execution for solving the linear system.

# the number of iterations per message size is cut down
# dynamically when a certain run time (per message size sample)
# is expected to be exceeded. Time limit is defined by variable
# "SECS_PER_SAMPLE" (=> IMB_settings.h)
# or through the flag => -time

# the number of iterations per message size is cut down
# dynamically when a certain run time (per message size sample)
# is expected to be exceeded. Time limit is defined by variable
# "SECS_PER_SAMPLE" (=> IMB_settings.h)
# or through the flag => -time

Job <mpirun --mca btl self,openib ./IMB-MPI1 pingpong> was submitted from host <411head> by user <root> in cluster <phpc_cluster>.
Job was executed on host(s) <1*compute004>, in queue <medium_priority>, as user <root> in cluster <phpc_cluster>.
<1*compute005>
</root> was used as the home directory.
</shared/compile_temp/intel/IMB/imb/3.2.4/src> was used as the working directory.
Started at Thu Oct 10 13:53:07 2013
Results reported at Thu Oct 10 13:53:09 2013

# the number of iterations per message size is cut down
# dynamically when a certain run time (per message size sample)
# is expected to be exceeded. Time limit is defined by variable
# "SECS_PER_SAMPLE" (=> IMB_settings.h)
# or through the flag => -time