Postfix Performance Results

Note: in the USENET news article below I am still referring to
the Postfix project by its obsolete MX code name. In addition,
some people pointed out to me that Qmail should be spelled as qmail.
I stand corrected.

Thought you might be interested. MX is the code name of the freeware
mailer project that I've been working on for the last few months
while visiting IBM TJ Watson Research.

Initial results are promising: MX is up to three times faster than
its nearest competitor, Qmail. Discussion follows below the
signature; tables are at the end of this message.

The mailer is based on a dozen semi-resident daemon programs, almost
all of which can run inside a chroot/nobody jail. I'm using tricks
from web servers to reduce fork/exec overhead. And, just like web
servers, my mailer supports real virtual hosting. You can give each
virtual host its own limit on, for example, the number of processes.

My mailer is designed to be sendmail compatible at the user level
so it supports /etc/aliases, NIS aliases, ~/.forward files and can
deliver to /var[/spool]/mail. However, it has no sendmail.cf file,
and none of its constituent programs is set-uid.

MX is the current project code name. I understand that this name has
already been taken by a VMS mail product. Suggestions for a better
name are welcome. The best suggestion so far: Blackmail, suggested
by Dan Farmer :-).

First release to the net about two months from now. I'll try to
set up a fact sheet web page as soon as I have switched to the
mailer myself.

Let there be some friendly competition.

Wietse

This week my mailer finally became complete enough that I could do
some first performance tests. I wrote a few tools to generate and
receive arbitrary amounts of SMTP messages and did some measurements
on my local 10MHz home ethernet. For a first experiment I used my
tools to compare Qmail performance to that of my own MX mailer.

The setup involved a message generating program, the mail software
under test, and a message sink program.

The message generator and sink programs ran on BSD/OS 2.1, P166.
The mail systems ran on FreeBSD 2.2.1, P133 Thinkpad. When I used
the generator and sink programs with no mail system in-between,
these two machines were able to pump 250 SMTP messages per second
over the 10MHz ethernet. Of course no real mail system can achieve
that, it just shows that the overhead of the generator and sink
tools is small compared to the overhead of Qmail and MX.

One test could not be completed because my MX mailer saturated the
message sink machine. The P166 is my main machine and has to stay
up, so I was unable to reconfigure it for a larger network load.

About the only test where Qmail performance approaches that of MX
is when sending one single message to a lot of different places:
76 deliveries per second (see tables below). Note that this is
still far below the 250 messages/second that I get when my message
generator and sink talk to each other over ethernet without mail
system in-between. Some speed-up may still be possible.

However, Qmail is three times slower with delivery of messages that
have only one (local or remote) recipient. With one recipient per
message, MX does 5.3 remote deliveries/second (4.2 local/second)
while Qmail does only 1.8 remote deliveries/second (1.7 local/second).

With the latter tests, speed is dominated completely by disk I/O.
It is obvious that Qmail hits the disk much harder than MX does.
Fixing that would require a complete redesign of Qmail's database
structure: Qmail uses three files per mail message, whereas my
mailer uses only one. This explains the factor of three in speed.
All other differences in architecture vanish in the face of this
disk I/O bottleneck.

(*) message sink host saturated under the load :-(
Local delivery performance: time in seconds to deliver 100 incoming
SMTP messages to one local recipient. Inbound SMTP concurrency =
5, 10 and 20 (number of incoming messages being pumped into the
system simultaneously).