Dr Manhattan has asked for the
wisdom of the Perl Monks concerning the following question:

Hi all

I have a text file that I want to extract some information from, however the file is too large to read all at once. So at the moment I'm trying to read in 3000 lines at a time, processing and extracting info, printing it, clearing memory and then go on to the next 3000 lines.

Looks like @array’s memory is never cleared. Also, you don’t need to clear @information explicitly — it’s a lexical variable, so will be re-initialised each time the if condition is true. So, change that line to:

This would be an ideal situation in which to use the built-in module Tie::File. That won't suffer from memory issues due to the size of the input file and would allow you to eliminate the need for the while loop, chomp, push, if condition and $counter. Also, you don't appear to be storing data in @information for subsequent use so you can eliminate that variable and the for loop that processes it. Here's roughly what you'd need:

Others have pointed out the real problem with your code but I would like to point out that with most IO architectures, you're probably not going to gain much performance by buffering your input this way -- the underlying library calls for read are probably already buffering. You may want to Benchmark your buffering approach with a standard line-by-line approach. If the differences are minimal, I would opt for the simpler code.

Why don't you simply read one line at a time, process it, print out what you need to output, and then go to the next line? FH iterators are great, and input buffering is done under the surface anyway (unless you take steps to prevent it).

I can't actually understand why you're trying to read line by line, and then batch process every 3000. Are the data in those 3000 lines in some way correlated? Otherwise, you're not really doing much good - a 'while' loop will do what you want without needing to buffer anything.