Optimize exim performance

I experienced that exim was slow in sending outgoing mail, it was spending a lot of time in the queue, resulting in customer complains. So I started to google what could be done to increase performance in exim. I didn’t have much knowledge about exim, and I needed a quick fix to get the mail queue going. I googled, and even though there was a myriad of information about exim configuration options which were poorly explained, I didn’t find the golden answer I needed.

So I spent a week researching, reading and at some point I got it.

There are many ways to increase performance depending upon the servers need. I had a server and its sole purpose was to receive and send thousands of mails (news letters) during a short time period. I wanted as little mail to be stuck in the mail queue as possible.

Whenever I mention values on options, you have to choose the one that best suits your server. To figure this out, the best option is to make configuration changes, and monitor the performance impact it has on your server. How you can do that is written in the bottom of this article.

One of the performance options is related to how the mail is built up. Does the mail only have one recipient or multiple recipients?

Setting number of mail queue processes

By default exim spawned off one mail queue process (runner) every hour which was way to little. I needed multiple runners to receive and send the mail immediately without delay. This option is particularly useful when mails only have one recipient, but every exim server should configure this option anyway.

Change how many queue runners that are spawned off. In /etc/init.d/exim I changed this option: QUEUE=5m. This ensured that a new queue runner was created every 5 minutes. How often a new queue runner needs to be configured on a server depends on its load.

Change max number of queue runners that can exist simultaneously in /etc/exim.conf: queue_run_max = 50

Since we made changes in /etc/init.d/exim, we need to restart the server to make the effects off number of queue runners: /etc/init.d/exim restart

Multiple recipients
If a message shall be delivered to multiple remote hosts (multiple recipients), exim can be configured to run several smtp deliveries simultaneously by setting the configuration option “remote_max_parallel” to a value greater than one. Otherwise a delivery is sent serially. Max number of simultaneous delivery processes is then “queue_run_max * remote_max_parallel”.

In /etc/exim.conf change/add following: remote_max_parallel = 50

Whenever making a change to exim.conf, run the following command to ensure that exim reads the configuration changes and puts them in effect:
kill -HUP `cat /var/spool/exim/exim-daemon.pid`

Message management

There are multiple configuration options in how exim shall deal with the connections/mails itself, and these are the options I have played with.

#Simultaneous number of incoming connections to accept.
smtp_accept_max = 500

The server I was configuring had trusted servers that connected to exim and pushed thousands of mails per connection. So this option is a nice one:
#How many messages to accept per connection
smtp_accept_queue_per_connection = 2000

There is a point where the load gets so high that the server is unable to deal with what is coming. What that limit is depends on the hardware, software running on it, you name it. So the limit is different for every server. When the limit is reached, that’s when the server stops responding. You can set exim to have no queue deliveries if load is high in /etc/exim.conf: deliver_queue_load_max = 10

How to monitor the server during high load

The top command can be used to view the current load on the server. In the picture you can see the load average in the top right corner (1, 5 and 15 minute average). I experienced that during high load, the server had an load average up to 30, but it barely registered in the cpu. That told me that I could safely configure the server to many more queue runners etc. because it was able to handle the load. I will not even try to enter the science of load average which I have yet to understand. That’s candy for another time.

The sar command can be used to view historical load on the server by day, hour, minutes: /var/log/sa. Check the date on the files. sar04 is the 4th of the month in question. Command:
sar -f sa09 | less

A little confused about the difference between queue runners and SMTP deliveries — are they the same thing? For example, you say: “Change max number of queue runners that can exist simultaneously in /etc/exim.conf: queue_run_max = 50” and above that, “In /etc/init.d/exim I changed this option: QUEUE=5m. This ensured that a new queue runner was created every 5 minutes. ”

So one queue runner every 5 mintus equals 12 in an hour, yet you also say queue_run_max = 50 so it seems you would never hit that limit.

I interpret the queue_run_max as a failsafe or a way to ensure that your system isn’t maxed out. I have yet to find info on when queue runners die. I have seen many defunct queue runners that quickly dies, but I don’t know what makes the queue runner die in the first place. So what your limit should be on queue runner depends on your system and what it can handle.

Wow that was strange. I just wrote an extremely long comment but after I clicked submit my comment didn’t show up. Grrrr… well I’m not writing all that over
again. Anyway, just wanted to say fantastic blog!

One question: you send all these emails by php script, or any PC software?

Do you know any script that can be used for sending these emails? I am looking for someone who has smtp authentication and that can open more than just ONE simultaneous connections, like Interspire Email Marketer, for example. By the way, the IEM is a good script to send e-mails, but the problem is that only open 1 smtp connection, and it is very slow!

I tried to add queue_run_max and when I restarted exim4 I got this error:root@ds6419:/etc/exim4# /etc/init.d/exim4 restart
[....] Stopping MTA for restart:/usr/sbin/update-exim4.conf: 36: /etc/exim4/update-exim4.conf.conf: queue_run_max: not found

Why would ‘queue_run_max’ not work?
I run a split configuration file, so I add it to update-exim4.conf.conf with my other exim4 configuration settings.

I also have a split configuration. The file you need to edit in your case is most likely in /etc/exim4/exim.conf.template as well as /etc/exim4/exim.conf (if it exists). In my case lines such as queue_max_run and remote_max_parallel did not exist anywhere in the file so just add them and restart exim and you should be good to go. A tip to help you check if your changes have been made is run this command after restarting: exim -bP

@haridasi

Fantastic article. It really helped me. My config files didn’t have the lines you mention in your article so it took me ages to figure out I had to add them myself 😀