In my last article about cluster performance, I found that a cluster performs better than a single Pi, but there was a still lot of room for improvement. I've made changes to Apache's configuration files, and I've modified the way page caching works in my CMS.

Moving the page cache

The CMS that I've written can generate pages dynamically, and it can cache pages so that they can be served instantly without having to be assembled. Only pages that consist of static HTML can be cached. Pages that contain dynamic content generated by executable scripts aren't cached.

The page cache used to be in /usr/share/cms/cache. The Python interpreter had to be loaded to serve cached pages from /usr/share/cms/cache. Now, the root directory of the page cache is /var/www, so Apache can serve cached pages without invoking Python to run the CMS script.

One downside is that the CMS can not count track traffic anymore. When the CMS executed every time a page was requested, a function was called to increment a number in a data file. This doesn't work now that pages can be served without the CMS executing.

Unload unused modules

One of the best ways to improve Apache's performance is by unloading modules that aren't needed. First you need to list all the modules that are currently loaded using this command:

apache2ctl -M

It can be difficult to determine which modules are in use. It really depends on which directives are used in .htaccess files and virtual host files. For example, if you disable authz_host_module, then Allow, Order and Deny directives won't work. Each time you disable a module, restart Apache with these commands:

$ sudo service apache2 stop
$ sudo service apache2 start

You can use 'restart' instead of 'start' and 'stop', but there are some variables that require Apache to be stopped before they can be updated. It's a good idea to thoroughly test your site before you disable any more modules. I disabled these modules:

If you find that a module is required, you can re-enable it with the a2enmod command like this:

$ sudo a2enmod authz_host

Lower the timeout

I set the timeout in /etc/apache2/apache2.conf to 30. This prevents concurrent requests from occupying memory for long periods and reduces memory usage.

Tune Apache processes

Apache has several different multiprocessing models. They each use a number of server processes and child threads to handle HTTP requests.
MPM worker is the most modern configuration. MPM prefork used to be the standard, but MPM worker gives better performance and memory usage. Use this command to check what mode Apache is in:

The size of the Apache processes varies depending on the content being served and any scripts that might be running. This command shows the number of Apache processes, and their size:

ps aux | grep 'apache2'

The 6th column contains the amount of memory used by each process. Dividing the amount of spare memory by the size of the average Apache process gives a rough indication of the maximum number of server processes that you can run. Each Pi in my cluster has about 280MB of RAM that's free, and the average size of Apache processes is about 7MB. 280 divided by 7 gives 40.

StartServers is the number of server threads that Apache creates when it starts up. Creating new server processes can be time consuming, so I want Apache to start a lot server processes when it starts. This means it won't have to spend time creating more processes while it's busy processing a lot of traffic. I've set StartServers to 40.

I don't want Apache to be able to create too many processes, as my Pi might run out of memory, so I've set the ServerLimit to 40.

Each server process can have a varying number of threads. It's the threads that actually process requests. I've set the default number of threads per child to 8. I didn't calculate this, I just tried a lot of different numbers and ran a lot of tests with siege until I found the optimum value.

The total number of threads is the number of server processes multiplied by ThreadsPerChild, which is 320 with my settings. I set MaxClients to 320 to prevent Apache from creating extra threads.

These settings will cause Apache to create a lot of processes and threads that don't get used immediately. In order to prevent Apache from deleting them, I set MaxSpareThreads to 320.

MaxRequestsPerChild is the number of requests a process should handle before it is killed and a replacement process is started. This is done to prevent memory leaks from accumulating large amounts of memory. It should be set to the number of hits a server gets in a day so that processes are restarted once a day.

After optimizing page caching, removing unused modules from Apache and tuning the server process, the number of transactions per second for a single node has gone from 0.5 to over a hundred. The number of conncurrent requests that can be handled has increased by a factor of 8.

Tuning Apache processes has resulted in a very small decrease in the number of transactions per second, but the longest transaction time has decreased considerably.

Testing the whole cluster

Once I was happy with my new settings, I flushed them through to the entire cluster. I ran more tests with siege, first with 200 concurrent users:

Before tuning Apache, the cluster could handle 350 concurrent requests, and the maximum transaction rate was 460 transactions per second. Now the maximum number of concurrent users with 100% success rate is 800. The maximum number of transactions per second is now 947.

I will carefully watch the amount of spare memory over the next few days. If it starts to get too low, I'll reduce some of these settings.

Using siege isn't completely realistic. A request from a browser puts a much larger load on a server than a request from siege, and the timing of tests in siege is different from the ebb and flow of real traffic. Tests done using siege do not predict the number of requests a server can handle. Tests like these do give a basis for comparison of different server configurations. I don't think my cluster can handle 947 actual visitors per second, but I'm confident that my server's performance is better than it was.