Thursday, February 20, 2014

Use of threading in mod_wsgi daemon mode.

Every so often I get asked why when examining a mod_wsgi daemon process do there appear to be more threads running than what have been configured to handle requests. Since I can never be bothered trying to find what I wrote the previous time, I end up writing the explanation from scratch each time. Figured it may be time to simply throw it up here on my blog so I can at least refer to it here.

What this results in is mod_wsgi creating a set of three processes distinct from the main Apache child worker process. Multiple instances of the WSGI application will then be run, one in each process. The Apache child worker processes will automatically proxy the requests, with a request being picked up by one of the daemon processes and handed off to the WSGI application to handle. With the number of threads being specified as two, the most concurrent requests each process can handle is two. With a total of three processes, that is a total of six concurrent requests able to be handled across the whole WSGI application.

If we were to now look at the resulting process tree using something like htop tree view, we would see:

The question is, since we have configured the daemon process group to only have 2 threads per process, why are we seeing a total of 5 threads in each process?

The answer is that for a configuration of:

WSGIDaemonProcess mysite threads=num ...

there will be num+3 threads, where 'num' is the number of request threads indicated by the 'threads' option to the WSGIDaemonProcess directive. If no 'threads' option was specified, then the number of request threads will be 15.

What are these three extra threads then? They are as follows:

1. The main thread which was left running after the daemon process forked from Apache. It is from this thread that the requests threads are created initially. It will also create the other two additional threads, the purpose of which is described below. After it has done this, this main thread becomes a caretaker for the whole process. It will wait on a special socketpair, which a signal handler will write a character to as a flag that the process should shutdown. In other words, this main thread just sits there and stops the process from exiting until told to.

2. The second thread is a monitoring thread. What it does is manage things like the process activity timeout and shutdown timeout. If either of those timeouts occur it will send a signal to the same process (ie., itself), to trigger shutdown of the process.

3. The third thread is yet another monitoring thread, but one which specifically detects whether the whole Python interpreter itself gets into a complete deadlock and stops doing anything. If this is detected it will again send a signal to the same process to trigger a shutdown.

So the additional threads are to manage process shutdown and ensure the process is still alive and doing stuff.

Now under normal circumstances there will be one further additional thread created, but this is a transient thread which is only created at the time that the main thread has detected that the process is due to shutdown. This thread is what is called the reaper thread. All it will do is sleep for a specified period and if the process is still running at that point, it will forcibly kill the process.

The reaper thread is needed because the main thread will provide a short grace period to allow requests to complete and then destroy the Python interpreter, including triggering of any 'atexit' registered callbacks in each sub interpreter context. Because pending requests and destruction of the interpreters could take an unknown amount of time, or even hang, the reaper thread ensures the process is still killed if shutdown takes longer than allowed.

I am just saying that if no 'threads' option was given to WSGIDaemonProcess directive, then 'num', the number of request threads for each process defaults to 15. In that case it would be 15+3 threads, or 18 active threads in total in each process if you were too look at what was running. The number of process doesn't come into it as far as how many threads are running in a specific process.