Supervisor with Django and Gunicorn

This post assumes that you have used gunicorn and know what it does. I will try everything inside a virtual environment and hope you do the same.

What is supervisor.

Supervisor is a monitoring tool that can monitor your processes. It can restart the process if the process dies or gets killed for some reason.

Use of supervisor: Why I started using it.

In production, I use gunicorn as web server. I started a gunicorn process as a daemon and logged out from the server. My site ran as expected for few days. All of a sudden, we started getting ‘502 Bad Gateway’ and I had no idea why. I had to ssh to the server to find out what went wrong. After ps aux | grep gunicorn, I found out gunicorn wasn’t running anymore. My gunicorn process died on its own, and I had no idea when and why. Had I used supervisor, supervisor would have been controlling the gunicorn process. It must have recieved a signal when gunicorn died and it would have created a new gunicorn process in such scenario. And my site would have kept running as expected.

Other scenario

We want to run a process which doesn’t allow deamonizing. eg: I wanted to keep celery, a Python worker, running on the production server. I could not run it in the foreground because I had to logout from the server. I did not find an easy way to run celery as a daemon. Here too, supervisor came handy.

Project setup

I am setting up a new Django project so you will have proper idea of the path and how to use paths in supervisor configuration file.

Make sure you can still see /admin/. You will not be able to view static files now, because we aren’t using Django development server anymore. Read this post if you want to know how to serve static files using nginx. Though you don’t need to do this now, our motive is not to see static files.

Let’s run gunicorn as a daemon. Also, we will tell gunicorn to use a file to keep track of process id, so that we can use that process id to kill gunicorn whenever we want.

The problem with this approach is, gunicorn might die at any moment and you will not know about it immediately. Since we are working on dev environment, it is acceptable but gunicorn dying on a server is not acceptable. Your site will loose too many users/customers.

Before supervisord will do anything useful for you, you’ll need to add at least one program section to its configuration. The program section will define a program that is run and managed when you invoke the supervisord command. To add a program, you’ll need to edit the supervisord.conf file.

So, we’ll add a program section for gunicorn to supervisord.conf. Add it after the [supervisord] section.

You need to change command and directory depending on your environment. My gunicorn executable resides at /home/akshar/.virtualenvs/PythonEnv/bin/gunicorn. Find where your gunicorn executable is and use that in command section of program.

We passed testproject.wsgi:application as application object to gunicorn, so testproject need to be available on the PATH for this to work. And for something to be available on the path, you need to be in correct directory. That’s why we need the directory setting inside program section.

Now a supervisor process as well as a gunicorn process should be running. Even though you did not explicitly start a gunicorn process, supervisor started gunicorn for you. Verify that supervisor as well as gunicorn process exists. You need to check these on a different terminal, because supervisord must be running on your first terminal in the foreground, since it wasn’t daemonized.

You need to restart supervisor to see these changes taking effect. Kill the existing supervisor process by checking its id from supervisor pid file. Notice that you have a pidfile option under your supervisord section, which tells the file which stores process id of supervisor.

Surprise

Now gunicorn should be killed and you should not be able to access /admin/. But you will be able to access it, try it. Surprised? Check the pid of gunicorn now.

(PythonEnv)/tmp/testproj $ cat /tmp/gunicorn.pid
10301

So, somehow a new gunicorn process was created as soon as we killed gunicorn, and the process id for that gunicorn process was written in /tmp/gunicorn.pid.

Why did it happen.

When you killed gunicorn, supervisor came to know that gunicorn process died. And it’s supervisor’s duty to restart any process if it dies, if supervisor controls that process. Since gunicorn was being controlled by supervisor, supervisor made sure to restart gunicorn. So, be assured that if you are using supervisor and let it control gunicorn, then supervisor will make sure to always keep gunicorn running. And your site will never be down.

This is how we used supervisor on our development environment. You can similarly use supervisor on a production server.

It is worth mentioning that supervisor conf can have multiple program section. So, you can use single supervisor instance to control gunicorn, celery and any other process you want.

Thank you for reading the Agiliq blog. This article was written by akshar on May 9, 2014 in
supervisor
.