Currently I am working with some large data (most of it generated by Mathematica itself). I usually find it a hassle to do this. For example, I just exported a large amount of data to WDX on a machine with a lot of memory, just to find that I can't read it on my own machine (with little memory) because the file can only be read as a whole. It is also extremely slow to import (but using MX was not an option due to different architectures)

Mathematica is excellent when working with in-memory data, as it's paradigms of operating on data as a whole (Map, Outer, Table, etc.) are very convenient. But it is not great at working with data that is too large to fit into memory, and it is not good at sequential processing of on-disk data.

There have been discussions about this (see the comment discussions on this and this question), and the following idea came up more than once: it would be great to be able to use Mathematica's native paradigms to work on large on-disk data. The data would be loaded on-demand from disk, and discarded when not needed any more.

I'd love to hear some ideas on how to implement a framework that does this, but read the next section for a more practical question.

Question

How can I conveniently work with data that doesn't fit in memory? Can we implement a list-like structure which fetches the data from disk as needed? (For example, when indexed, it would load only the requested list-item directly from disk. As processing of this item has finished, the memory it took up would be freed.) Alternatively could we implement variable which are loaded from disk on demand, but can be unloaded from memory?

I'd prefer not to have to deal with things like file names (if this is backed my multiple files). I'm hoping to be able to have a nice abstraction, where I never need to do explicit reading from disk. I'd like to work with an object which acts and works similar to an in-memory list.

Ideas

This could be backed by the MX format, which is very fast to read and can store any expression. Unfortunately it's not portable between machines. For machine numbers, a flat binary file and BinaryReadList could be useful.

Personally I find life so much easier by having data in a database and linking to Mma.
–
Mike HoneychurchJan 17 '12 at 23:11

@Mike I have never done that, it's good to hear experiences in how well that works. E.g. how long would it take to load 2 GB of data into Mathematica, compared to MX files?
–
SzabolcsJan 17 '12 at 23:12

@Szabolcs I haven't worked with stuff that large and I would imagine that you will run into Mma limitations. From your background and question I thought you only wanted to bring into Mma "chunks" of data on demand. In other words do you really need an entire 2GB or can you do some SQL operations to pick out what you need?
–
Mike HoneychurchJan 17 '12 at 23:20

1

@MikeB that doesn't scale though, and is inconvenient. I generally do the same (having access to machines with 512GB of RAM helps), but I'd like to know how to do things in a more reasonable way.
–
aclJan 18 '12 at 1:18

1 Answer
1

Preamble

I spent some time and designed and implemented a tiny framework to deal with this problem, over the last two days. Here is what I've got. The main ideas will involve implementing a simple key-value store in Mathematica based on a file system, heavy use and automatic generation of UpValues, some OOP - inspired ideas, Compress, and a few other things. Those who know my posts, I have to warn that this is going to be an unusually long one.

The problem and ideas behind the solution

Let me describe the limitations of my system right away. Since the general problem is tough, I consider a very simplified version, but one which can be useful in its own right, and which can serve as a good starting point for future developments. The problem is how to file-back a large ragged numerical list, whose sublists are possibly packed, but generally of different lengths. Let me tell from the start that since I can not use .mx files to avoid platform-dependence, the performance of this won't be stellar. This is a clear speed/memory trade-off situation, and the performance will be merely average. Perhaps, one could make a few tweaks. The overall design was more of my concern here, and I hope I've got a few things right in that department.

Let us say we have a large list already constructed in memory in Mathematica, call it testList. Its elements are lists themselves. What I will do is traverse it element by element. For a given element (sub-list), we will analyze how much memory it occupies, and if this amount exceeds a certain threshold that we specify, we will create a key-value pair for it. The key will be some dummy generated symbol, and the value will be a file name for a file where we will save a contents of this element. We will actually Compress the element first, and save the compressed data.

Low-level OOP-style data exchange API

EDIT

Since using .mx files is so much faster, I added some switch functions which will allow one to switch between using usual files and .mx files:

In addition, compression / uncompression we will also be able to switch on and off. Note also that other functions down the page have been modified accordingly.

END EDIT

As a second component, we need some high-level structure, which will represent the "skeleton" of the original list, and which will manage the on-demand data fetching and saving. As such a structure, I will use just a single symbol, say s. Here is the function which implements the management (the large one):

How it works

Let me now explain what happens here. First, LetL is a sequentially-binding version of With, which I will display in a minute. It allows to avoid nested With statements. The parameters of the function are the main top-level symbol s, the part index, and the directory where our key-value store will be located. Basically, in OO terms, this function creates an instance of a class, with these methods: Part (part extraction), releasePart (releasing the memory occupied by the part, and getting ready to extract it from file again, savedOnDisk - checks is the part has been backed into a file, removePartOnDisk - deletes the backing file for the part, savePartOnDisk - save the part contents to a file, and releaseAPI - needed to release resources at the end.

All this is implemented via UpValues for s. In particular, the Part is overloaded, so now when I call s[[part]], it will look and feel like I extracted the part of s (not true of course, but very convenient). The content of the part is stored in the generated symbol sym, which is unique for a given part. Notice that the definition is lazy and self-uncompressing. This is a similar technique to one I used in this answer. Upon the first call, sym loads the content from file and uncompresses it, and then assigns it to itself. All subsequent calls will be constant time, with the content of the part stored in sym. Note also that when I call releasePart, I remove the direct part content from sym, feed it to the garbage collector, and reconstruct back the lazy definition for sym. This is my mechanism to be able to release part content when no longer needed, but also be able to load it back again on demand.

There are two important points to note regarding Compress. One is that it does not unpack packed arrays. Another is that it is cross-platform. Both are huge wins for us. Note that, essentially, for each part I create an instance of a class, where sym plays a role of instance variable. Note also that I use the Hash of the name of sym, to construct the file name. There are two flaws with this approach actually. One is that there in principle can be hash collisions, and currently I don't handle them at all. Another is that the symbols sym are unique only within a single session, while, as we'll see, I will be exporting their definitions. Both problems are surmountable, but for the sake of simplicity, I ignore them for now. So, the above code represents the low-level data-exchange API on the level of a single list's part.

Higher-level interface: the list-building function

This is the main function used in list-building. Its name pretty much tells what it does - it extends the list with one more element. This, however, does not cost us a performance penalty, since our "list" is faked - it is a symbol s which pretends to be a list but in fact is not (it is more like a hash-table filled with class instances).

As you can see from this code, not all parts of the list are backed by files. Those which are below the threshold in terms of size, are merely compressed and also assigned to s via UpValues and overloaded Part, but are not on the disk. The code of this function is pretty self-explanatory, so I will move on.

Integration with the system and initialization

The following function (partially) integrates my construction with some commands that we all love. This will help to better masquerade our symbol s so that in many respects it now behaves as an ordinary list.

The last several functions are concerned with disk management, and storing the main structure / definitions on disk. The point is that in the process of creating our key-value store, we generated lots of UpValues for s, and all those private symbols sym for each part, must also be saved together with s, if we want to fully reconstruct the environment on a fresh kernel.

This will find the dependencies of the main symbol s. We only use UpValues, so this is quite straightforward.

There are a few subtleties here. The problem is that Save converts delayed UpValue definitions (made with TagSetDelayed or UpSetDelayed), into immediate ones (which looks like a bug to me, but anyways). Therefore, I have to load the package in unevaluated form and do back replacements manually, before I allow it to run.

The last function here will completely remove all the generated files from the file system:

Need for speed: Turning on .mx files

If we sacrifice being cross-platform for speed, we get 10-40x speedup by using .mx files, and in this regime I'll be hard-pressed to see any database solution beating this in terms of performance. Here are the same benchmarks as before, done with .mx files.

Summary and conclusions

I presented here a tiny but complete implementation of a key-value store, which may make it possible to work with large files which don't fit in memory, notably lists. From the technical viewpoint, this is by far the most serious application of UpValues I have ever written. I think the simplicity of the code illustrates the power of UpValues well. They also made it possible to have nice syntactic sugar, and be able to use the familiar commands such as Part, Take, etc.

The implementation has many flaws, and it is still not clear to me whether it is efficient enough to be useful, but I think this may represent a good starting point.

EDIT

As it turns out, using .mx files gives a huge speedup (which is not unexpected of course). If speed is absolutely crucial, one can use .mx files for all computations and only use normal files to import from or export to another computer. I plan to build a layer which would automate that, but so far, this can be done manually, based on the single-part API in the code above.

Wow Leonid! Many thanks for this! I won't read it tonight (some more work to do), but it'll be the first thing in the morning, with a fresh mind.
–
SzabolcsJan 18 '12 at 20:40

1

@Szabolcs This had to be done. We were ignoring this problem for too long.
–
Leonid ShifrinJan 18 '12 at 20:46

It's a really good time to do it because right now I need it (this was a practical problem too right now)
–
SzabolcsJan 18 '12 at 20:47

@Szabolcs Well, may be you will be able to start using this right away! I really hope so! I am not quite sure about efficiency of this stuff - how acceptable it is. Man, I was thinking all night about how to do it, but only came to this idea several hours ago.
–
Leonid ShifrinJan 18 '12 at 20:50

1

@telefunkenvf14 Yes, I was actually lucky to have attended that talk in person. Very impressive, but Sal was more talking about how he was generating K by Mathematica - so some knowledge of K / KDB would be needed. Re: Hadoop link: so far I only know that it exists, and in fact has been open-sourced on GitHub IIRC. But I may have a chance to get to know it much better in the near future. Re: PostgreSQL: much slower than .mx files, I am sure about it (did not benchmark however), plus potentially my framework is more flexible, since the data in a list can be any Mathematica expression.
–
Leonid ShifrinNov 27 '12 at 18:43

Mathematica is a registered trademark of Wolfram Research, Inc. While the mark is used herein with the limited permission of Wolfram Research, Stack Exchange and this site disclaim all affiliation therewith.