Software development and daily life in Seattle

Menu

Django Queue Service

I know I’ve been quiet on the Django front for a while. Here’s something to make up for it. Without further ado, let me present to the world the Django Queue Service.

At OSCON 2007, I was hunting around for something to use to deal with background processing initiated from my web application. Not finding anything that I could immediately use and implement, I took it as a challenge to knock out something in the time I was there. The result is the django queue service.

It was later in the week (at the Django BOF) that I learned about TheSchwartz and Gearman, which Brad Fitzpatrick is hacking away it to make a queue mechanism available for fellow Python and Ruby geeks.

At the time, I thought “I wonder if I can hack this out in a week…” Well, yes I could. I modeled the service after the REST API from the Amazon Simple Queue Service, because that seemed both reasonable and effective. And I didn’t mind the idea of having my own Queue that (mostly) matched that API that I didn’t have to pay anything to use.

The project also contains from code from the CherryPy project. Mark Ramm (a really nice fellow, by the way) gave a session at OSCON about WSGI servers that was really intriguing. I’ve been looking for a “pure python” service to run Django with for a while, and after chatting a bit with him, thought I could probably make it happen with the CherryPy WSGI server. So I did. I have a little piece of linkage code that fires up the CherryPy WSGI server code and links it right on through into Django. I didn’t want to use the rest of CherryPy, so I nicked that code and dropped it into this project as a nice reference and a handy way to run the queue.

I suppose this project could easily end here, and of course there was the “but I want to…” list of things that I had in mind to do before I released it into the world. Well – as they saying goes: “Now is better than never.” so out it goes.

If you are interested in getting involved with the project, sending me patches, or whatever – let me know. You can leave a comment here or use that spiffy Google Code Hosting issue tracker. The project is functional, but not brilliant – an not entirely cleaned up in terms of its API. But enough excuses – its there, feel free to use it or not as you like, contribute or not as you like.

3 thoughts on “Django Queue Service”

Right now I’m using it to start longer running background processes scanning machines in a data center to get updated metrics and associated information. The results are tossed into the database (using Django, naturally), along with a last updated time stamp – so I can see what’s happened and when.

I am new to Python, But when I was supposed to run those longer processes using Django, I used threading module…used to make threads for each request. and update the page with the result using JSON.
So using Django Queue Service would I be able to run and enqueue processes??