If you need to modify a flat-file, and the size is such that it can't be slurped into memory (and a 1GB file probably can be ...), it's best to process the file one record at a time writing the modified records to another file. Then, swap the file-names around. If you slurp into memory, run a performance monitor to see if the virtual-memory subsystem starts thrashing, because VM swapping activity is a serious source of "invisible" disk I/O. If you process a record at a time, memory won't be an issue but data movement (a gigabyte is read and written) will be. But a one-gigabyte file should be processed sequentially in seconds, not 30 minutes.

Comment on
Re: Need help to fine tune perl script to make it faster( currently taking more than 30 minutes)