Handling OutOfMemory Error

Hi,
I'm operating on a 600MB file to extract some data from it.
I'm using ArrayList and HashTable for storing the values extracted
from the file and manipulating the data obtained.
When I try executing the program I'm getting the following error:

Advertisements

On Sep 16, 8:49 am, ruds <> wrote:
> Hi,
> I'm operating on a 600MB file to extract some data from it.
> I'm using ArrayList and HashTable for storing the values extracted
> from the file and manipulating the data obtained.
> When I try executing the program I'm getting the following error:
>
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Unknown Source)
> at java.lang.AbstractStringBuilder.expandCapacity(Unknown
> Source)
> at java.lang.AbstractStringBuilder.append(Unknown Source)
> at java.lang.StringBuffer.append(Unknown Source)
> at
> Analysis_Summary_Variable.getdata(Analysis_Summary_Variable.java:61)
> at
> Analysis_Summary_Variable.main(Analysis_Summary_Variable.java:549)
>
> How should I increase the JVM's memory to handle such large amount of
> data?
> Is there any other way around it?

The -Xmx command line flag lets you set the heap size. See java.exe
docs dor details..

Advertisements

On Mon, 15 Sep 2008, ruds wrote:
> I'm operating on a 600MB file to extract some data from it. I'm using
> ArrayList and HashTable for storing the values extracted from the file
> and manipulating the data obtained. When I try executing the program I'm
> getting the following error:
>
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Unknown Source)
> at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source)
> at java.lang.AbstractStringBuilder.append(Unknown Source)
> at java.lang.StringBuffer.append(Unknown Source)
> at Analysis_Summary_Variable.getdata(Analysis_Summary_Variable.java:61)
> at Analysis_Summary_Variable.main(Analysis_Summary_Variable.java:549)
>
> How should I increase the JVM's memory to handle such large amount of
> data?

-Xmx, as others have mentioned.
> Is there any other way around it?

Firstly, the stacktrace there is from a StringBuffer which is trying to
expand itself. If you can work out ahead of time how big that StringBuffer
eventually needs to be, or even put a useful upper bound on it, then you
can create the StringBuffer with that much capacity in the first place,
which will avoid the need to expand it, and might avoid that failure. It
should certainly improve performance.

I'd also look at whether you need to use that buffer at all - you're not
doing something like reading the entire file line by line and putting it
in the buffer, are you? If you are, find an alternative! In general, a
great way to reduce memory use is to find ways of doing things
incrementally, so you don't need to have all your data in memory at once.
For instance, if you were adding up all the numbers in a file, you might
do:

make an empty list of numbers
for line in file:
parse the line to a number
put the number in the list
set the total to 0
for number in list:
add the number to the total
report the total

But you'd use far less memory like this:

set the total to 0
for line in file:
parse the line to a number
add the number to the total
report the total

That's an obvious and trivial and example, and i imagine your example does
not admit such easy improvements. However, with the application of a
sufficient amount of cleverness, some degree of incrementalisation is
often possible.

Otherwise, whilst ArrayList is fine, the HashMap would worry me slightly
in terms of space. How are you using it? Is the set of keys the same or
similar between records? Could you use an object instead of a hashmap?

If you tell us more about your program, we can give you more specific
help.

tom

--
Any problem in computer science can be solved with another layer of
indirection. -- David Wheeler

Share This Page

Welcome to The Coding Forums!

Welcome to the Coding Forums, the place to chat about anything related to programming and coding languages.

Please join our friendly community by clicking the button below - it only takes a few seconds and is totally free. You'll be able to ask questions about coding or chat with the community and help others.
Sign up now!