Message Boards

Importing big files (ImportSequential)

If you ever tried importing a really big file, you presumably had to reboot your computer soon after, since your computer probably became unresponsive.

Recently I had to import a really big file (over 1Gb). For this, I created the code below.

It import line by line of your file (assumed to be a text file). Pretty straightforward.

It would be nice if Mathematica could check if the file size is too big to be imported by conventional methods instead of freezing the computer. (And it's cumbersome to wrap a TimeConstrained or MemoryConstrained over since we silly humans forget things easily...)

Indeed it would be interesting to add a parameter in the script to check the file size before importing it. If the file is too big an error message appears indicating to reduce the file otherwise the loading is done.

It is obviously useful to always remember that there are technical limits!

As for memory use, I think ReadList is the best choice right now. In principle, it should be possible to implement a more memory efficient reader in C provided that you know the datatypes in advance (e.g. all numeric, then use packed arrays).