For the most part, objects allocated by Django are short-lived, and are eligible for garbage collection when the request ends. In our case, we also have some long running jobs using celery. One in particular, a job to create a several hundred megabyte XML file, was consistently using all the RAM on the machine.

This wasn't too surprising because we were initially using Django templating to create the file, which keeps the entire response in memory while it's still being composed. But even after we moved to a SAX parser, which is specifically designed for running with little memory by streaming the file, we were still running out of memory occasionally.

We decided it was time to stop guessing, and profile the memory usage. Never having done this before, I did some research and came up with this excellent guide to using pdb for memory profiling.

To get started, I needed something I could run from the command line, outside of the celery task framework. Using the Django Command framework, I was easily able to compose a job that could run via manage.py.

I was excepting to see some SAX parser objects at the top of the list. Instead, most of the memory was tied up in Job objects, which are a Django model in my application. Looking at the function, jobs were first referenced in the following code block.

for job in jobs:
feed.write_entry(2, job)

Playing around a bit, I tried the following, which immediately solved the memory issue.

Basically, doing a separate query for each job, and making sure it goes out of scope after we're done using it. In retrospect, this was pretty obvious. So obvious, that Django even provides a handy idiom called .iterator() to do just this:

Evaluates the QuerySet (by performing the query) and returns an iterator (see PEP 234) over the results. A QuerySet typically caches its results internally so that repeated evaluations do not result in additional queries. In contrast, iterator() will read results directly, without doing any caching at the QuerySet level (internally, the default iterator calls iterator() and caches the return value). For a QuerySet which returns a large number of objects that you only need to access once, this can results in better performance and a significant reduction in memory.