I am running a Java program using Eclipse3.4.2. in Windows Vista.
I have set VM arguments :
-Xms512m
-Xmx1500m
After the program runs for some time, the error message pops-up:
Java.lang. OutOfMemory Error

My computer has 6GB of RAM but, if I try to increase the value of Xmx to
2GB,
it doesnt run and a message comes up:
Could not reserve enough space for object heap

I installed the Eclipse Memory Analyzer in the Eclipse using Software
Updates and Add-ons and tried to open the heap dump but then received the
message:
An out of memory error has occurred during parsing heap dump

it seems that you are running under 32 bit... The maximum amount of memory
that can be allocated under Windows 32-bit is roughly 1.4 - 1.5 GB (for
Java processes).

To get rid of the OOM during parsing the heapdump you should increase the
memory setting inside the eclipse.ini... In my installation eclipse 3.4.2
runs with
256 Mb... Try to replace the value of the -Xmx option with -Xmx1024m or a
higher value.

So I would recommend to set the MaxPermSize to 256m and also to decrease
-Xmx to 1024m. Try to load/parse the heapdump again... Normally these
settings work for all 32 bit heap dumps...

If you don't want to change your settings, I would recommend to download a
separate RCP of MAT and try to parse your dump with this separate
installation of MAT. (Here you have additionally the possibilty to parse
the heap dump within the command shell with: "ParseHeapDump.bat <path to
dump>".)

For a 6GB heap dump you will need a larger heap for the Memory Analyzer. You will need a 64 bit box. It is difficult to say how much heap you'll need to specify, as this depends on the number of objects in the heap and sometimes also on the shape of the object graph. From my experience you will need between 4GB and 6GB for the tool to analyze this.

On the download page there are RCPs for different 64 bit platforms. If you have no chance to run the Eclipse GUI, then you can run the ParseHeapDump script. Later you should be able to move the heap dump and all the generated .index files to a 32 bit box and analyze them there.