I've been starting to use Python's shelve module (and I'm new-ish to Python per se), and while it is storing and loading information perfectly, the RAM usage is... strange.

The parts I want to save is a map that's on certain coordinates of another map, and a selection of values like 'fov1' and 'fov2' which may or may not exist for any given coordinate. For instance, on "minimap" at x=50 and y=100, there may be another map, and I want that saving. Minimap[50][100] may also have, for instance, a list for fov0 and fov1, but not anything else. The max fov value is fov75.

Here are my save/load functions.

This save function is meant see which maps need saving (those which are not blank, and not currently saved), then to save the map as Xcoord + 'x' + Ycoord (eg '50x100') and save the fov parts as Xcoord + 'x' + Ycoord + 'f' + the number of the fov, (eg '50x100f2'). That all works fine. I also then want it to delete the map that's being held in RAM at that point, and to delete all the fov segments that exist so that the memory can be freed up. That also appears to work.

However, the weirdness is with RAM usage, as I've been watching task manager as I test it. Simply saving a single map actually INCREASES the RAM python is using! Surely that can't be right? Alternatively, if I save a large chunk at once, the RAM goes down as it should, but then if I load them all back up, the RAM is then significantly higher than it was before they were ever saved. If I then save/load again, the RAM doesn't go up; it only increases on the first save/load cycle, then never again. I've been working on this all day and cannot work out what is causing this strangeness, and how it can be resolved. Can anyone help me out?

Edit: oh, yes, using Python 2.7.2. Also, no more than 9 maps will ever need saving at once.

Unless your program is actually eating up all of your RAM, I wouldn't pay attention to fluctuations in how much it's using: if you really need that kind of control then you shouldn't be writing in python! Something I would be concerned with, however, is how you loop over all 76 p values for each y and x (that's potentially 3,040,000 different checks!). You should consider minimap[x][y] returning a dictionary of valid p values so that you can iterate over just your valid values. If you could post some more info on what you are trying to do I could be of more help.
–
Nolen RoyaltyApr 2 '12 at 23:31

Ah, that's not an issue; no more than a 3x3 grid of maps can need saving at once, so at the very most (and very rarely) that'll mean 684 checks! Basically, it's a map for a game - the map is a total of 40,000 x 40,000 tiles, split into a 200x200 set of 200x200 tiles. The minimap functions as a kind of world map, while each 'map', ie minimap[x][y].map, is a 200x200 chunk. When you move two chunks away from a map, that section and everything with it is unloaded (hopefully!). When you then move back towards it, it gets loaded.
–
UltimaRatioRegumApr 3 '12 at 16:25

1 Answer
1

The RAM usage as reported by Task Manager is not closely related to the actual working set of a process. It's important to keep in mind that memory management is handled at multiple levels. The OS delivers large chunks of memory into the process's heap, and in this case the Python runtime will allocate from, and free back to, the process heap, without returning significant amounts of memory to the OS (except in unusual situations).

The shelve module probably composes at least portions of the shelved version of the data in memory prior to writing them to disk, so that explains the initial increase.

If repeated reads and writes don't increase the total amount of memory in use, and you have enough memory to do what you're doing, there's nothing here to be concerned about.

That is interesting, and new knowledge to me. At the moment, things don't seem to going above an upper limit, but a small portion of the time, they still do. However, I think this problem may be due to the sequence of map creation/saving elsewhere in the code, and requires its own look at. Will update this tomorrow once I've had a look.
–
UltimaRatioRegumApr 3 '12 at 19:27