On Big Red II at IU, how do I use PCP to bundle multiple serial
jobs to run them in parallel?

Overview

Parallel Command Processor (PCP), developed by the Ohio
Supercomputer Center (OSC) and the
National Institute for Computational Sciences (NICS), is an application
that lets you bundle multiple serial jobs and run them
concurrently.

Using PCP to bundle and run your serial jobs in parallel lets you
make efficient use of all the cores on a compute node. Conversely,
running one serial job at a time can waste more than 90% of a node's
computational power. PCP also lets you request multiple nodes for your
jobs.

Running multiple serial jobs in parallel with PCP

For a parallel job with N processors allocated, the
PCP manager process will read the first N-1 commands in
the command stream and distribute them to the other N-1
processors. As processes complete, the PCP manager reads the next
process in the stream and runs it on an idle core.

Submit the script (e.g., pbs_script.sh) from the
command line using qsub:

qsub pbs_script.sh

As a result, the 31 serial jobs in list.txt will run
in parallel on Big Red II. When the PCP manager runs out of commands
to run, it will wait for any remaining running processes to complete,
and then shut itself down.

Getting help

Support for this system is provided by the UITS High Performance
Systems (HPS) and
Scientific Applications and Performance Tuning (SciAPT)
groups. If you have system-specific questions, contact the HPS
group. If you have questions about compilers, programming,
scientific/numerical libraries, or debuggers on this system, contact the
SciAPT group.