Reading the contents of a File without impacting heap space (Reading the file into memory?)

Reading the contents of a File without impacting heap space (Reading the file into memory?)

Hi all,

I am working with parsing the contents of a log file to check for the occurrence of a string (i.e. "CRITICAL alarm"). Was hoping for the application I'm using to be as light-weight as possible in terms of memory usage, etc. As a test, I created a 250 MB log file and noticed that I get a "java.lang.OutOfMemoryError: Java heap space" error when trying to deal with this big file. I replicated this by setting my maximum heap size to 128MB. Looks like its reading this file into memory, anyone know of any ways around this?

Re: Reading the contents of a File without impacting heap space (Reading the file into memory?)

Hi Dave,

Perhaps you should show us the code that you use for that task.

What I mean for example is that if you use something like new
File("log.txt").text or new File("log.txt").readLines(), then Groovy
will try to read everything in memory! But if you use something like
new File("log.txt").eachLine { ... }, it will read one line at a time,
and won't fill the memory.

> Hi all,
>
> I am working with parsing the contents of a log file to check for the
> occurrence of a string (i.e. "CRITICAL alarm"). Was hoping for the
> application I'm using to be as light-weight as possible in terms of memory
> usage, etc. As a test, I created a 250 MB log file and noticed that I get a
> "java.lang.OutOfMemoryError: Java heap space" error when trying to deal with
> this big file. I replicated this by setting my maximum heap size to 128MB.
> Looks like its reading this file into memory, anyone know of any ways around
> this?
>
> Thanks!
>
> Regards,
> Dave.
>
>

Re: Reading the contents of a File without impacting heap space (Reading the file into memory?)

Do you run your log analyzer in parallel (in background) with the real app generating the log? Anyway, I'd suggest 2 things.

1) Enable automatic log files rolling with limit as say 50 or 100 mb for app generating the log, it should be doable via configuration of logger (like Log4J or whatever you are using) - i.e. upon reaching this limit, the application starts new log file.

I am working with parsing the contents of a log file to check for the occurrence of a string (i.e. "CRITICAL alarm"). Was hoping for the application I'm using to be as light-weight as possible in terms of memory usage, etc. As a test, I created a 250 MB log file and noticed that I get a "java.lang.OutOfMemoryError: Java heap space" error when trying to deal with this big file. I replicated this by setting my maximum heap size to 128MB. Looks like its reading this file into memory, anyone know of any ways around this?

What I mean for example is that if you use something like new
File("log.txt").text or new File("log.txt").readLines(), then Groovy
will try to read everything in memory! But if you use something like
new File("log.txt").eachLine { ... }, it will read one line at a time,
and won't fill the memory.

Guillaume

On Wed, Jul 13, 2011 at 12:43, Dave McGee <[hidden email]> wrote:
> Hi all,
>
> I am working with parsing the contents of a log file to check for the
> occurrence of a string (i.e. "CRITICAL alarm"). Was hoping for the
> application I'm using to be as light-weight as possible in terms of memory
> usage, etc. As a test, I created a 250 MB log file and noticed that I get a
> "java.lang.OutOfMemoryError: Java heap space" error when trying to deal with
> this big file. I replicated this by setting my maximum heap size to 128MB.
> Looks like its reading this file into memory, anyone know of any ways around
> this?
>
> Thanks!
>
> Regards,
> Dave.
>
>

> Hi Guillaume,
>
> Thanks for the quick response! Apologies - I forgot to include my
> code. I think I am using the approach that you mentioned (new
> File("log.txt").eachLine { ... })
>
> File file = new File(/C:\Users\Dave\workspace\MyApplication\test.txt/)
>
> file.eachLine {line ->
> if (line.contains(id)) {
> alarmIdentified = 1
> }
> }
>
> This still seems to be giving an OutOfMemoryError. Can anyone spot
> anything I am doing incorrect?
>
> Regards,
> Dave.
>
> On 13 July 2011 03:48, Guillaume Laforge <[hidden email]> <mailto:[hidden email]>> wrote:
>
> Hi Dave,
>
> Perhaps you should show us the code that you use for that task.
>
> What I mean for example is that if you use something like new
> File("log.txt").text or new File("log.txt").readLines(), then Groovy
> will try to read everything in memory! But if you use something like
> new File("log.txt").eachLine { ... }, it will read one line at a time,
> and won't fill the memory.
>
> Guillaume
>
> On Wed, Jul 13, 2011 at 12:43, Dave McGee <[hidden email]> <mailto:[hidden email]>> wrote:
> > Hi all,
> >
> > I am working with parsing the contents of a log file to check
> for the
> > occurrence of a string (i.e. "CRITICAL alarm"). Was hoping for the
> > application I'm using to be as light-weight as possible in terms
> of memory
> > usage, etc. As a test, I created a 250 MB log file and noticed
> that I get a
> > "java.lang.OutOfMemoryError: Java heap space" error when trying
> to deal with
> > this big file. I replicated this by setting my maximum heap size
> to 128MB.
> > Looks like its reading this file into memory, anyone know of any
> ways around
> > this?
> >
> > Thanks!
> >
> > Regards,
> > Dave.
> >
> >
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

Re: Reading the contents of a File without impacting heap space (Reading the file into memory?)

Hi Daniel, all,

Good catch (and to John also!). I needed to add a "\n" to the end of the line on the log entry in the file I was generating for testing purposes. Looks like the eachLine was just going over a huge single line but I didn't spot it as it filled up my notepad window, which created the illusion of lines on random text! Schoolboy error!

What I mean for example is that if you use something like new
File("log.txt").text or new File("log.txt").readLines(), then Groovy
will try to read everything in memory! But if you use something like
new File("log.txt").eachLine { ... }, it will read one line at a time,
and won't fill the memory.

<mailto:[hidden email]>> wrote:
> Hi all,
>
> I am working with parsing the contents of a log file to check
for the
> occurrence of a string (i.e. "CRITICAL alarm"). Was hoping for the
> application I'm using to be as light-weight as possible in terms
of memory
> usage, etc. As a test, I created a 250 MB log file and noticed
that I get a
> "java.lang.OutOfMemoryError: Java heap space" error when trying
to deal with
> this big file. I replicated this by setting my maximum heap size
to 128MB.
> Looks like its reading this file into memory, anyone know of any
ways around
> this?
>
> Thanks!
>
> Regards,
> Dave.
>
>

---------------------------------------------------------------------
To unsubscribe from this list, please visit: