SBATCH logs errors after running newgrp

Submitting jobs from within a staging job

When you submit a job in a staging node, you get 8GB RAM allocated per core. If you would then submit a new job from within the staging node, this requirement is copied to the next job. If the next job is a regular job (e.g. in the normal partition), then this might cause issues. For example, if you would request 24 cores in this job, it will try to allocate 192GB RAM (8GB RAM each). However, a normal node has only 64GB in total. In that case, SLURM outputs the following error message:

This message is a cryptic error message: you are asking for 24 cores (192 GB) and we ‘have’ only 8 (64GB). In reality we do have 24 of course, so removing this memory setting should resolve the issue.

Advise: add a specific memory requirement X using #SBATCH --mem-per-cpu=X to the job-script or --mem-per-cpu=X as an argument to srun. This requirement is in MB and should be calculated based on the number of cores (e.g. 24) and the maximum memory available (e.g. 64GB). For example, X could be set to 2600 to use 2.6GB per core (24 * 2.6 = 64GB).