What we discovered is that our PBS mom daemon did not have unlimited locked
memory. So, when your job is created by the mom daemon it inherits the
memory limits. The fix was to cycle the PBS mom daemon after a boot (and
yes, we do start the mom daemon at boot but for some reason it doesn't
inherit unlimited locked memory). The way to determine if this is the
problem is to place a "ulimit -a" in the text of your PBS job. Run your job
and you will see a limit of 32K. Next cycle the mom daemon on the node(s)
of interest and re-run your job. You will now see unlimited memory.
Thanks,
Pat O'Bryant