Array Jobs

To run many instances of the same job, use the --array switch to
sbatch. This is useful if you have a lot of data-sets which you
want to process in the same way:

sbatch --array=from-to [other sbatch switches] YourScript

You can also put the --array switch in an #SBATCH line inside the
script. from and to are the first and last task number. Each
instance of YourScript can use the environment variable
$SLURM_ARRAY_TASK_ID for selecting which data set to use, etc. (The
queue system calls the instances "array tasks".) For instance:

sbatch --array=1-100 MyScript

will run 100 instances of MyScript, setting the environment variable
$SLURM_ARRAY_TASK_ID to 1, 2, ..., 100 in turn.

It is possible to specify the task ids in other ways than from-to:
it can be a single number, a range (from-to), a range with a step
size (from-to:step), or a comma separated list of these. Finally,
adding %max at the end of the specification puts a limit on how many
tasks will be allowed to run at the same time. A couple of examples: