Make sure to check if you are running JBoss with 32 bit JDK as 32 bit JDK has a process limitation of 4gb. If you are running JBoss with 32bit JDK, switch to 64bit which does not have a process size limitation of 4gb.