Incoming SMTP Mail

Now that we have some of that light background on the philosophy of
qmail, let's see how it expresses itself through an incoming email.

qmail, like most email systems, is all about moving email into and out
of various queues for delivery. qmail has one queue where it receives all
incoming email to be analyzed for delivery. Email is injected into that
queue via a program called qmail-queue. True to the DJB
philosophy, that program insures that email properly gets into the
incoming queue safely and securely. It will not respond with a success
code unless every step went properly. It is the responsibility of other
small programs to receive mail via SMTP, QMQP, or the command line, and
then properly feed the mail to qmail-queue. For our case,
we're only concerned with the SMTP handler, since that is how systems
receive mail from the internet. The specific qmail program that handles an
SMTP transaction is qmail-smtpd.

Since qmail programs are designed to do the minimum necessary, the
qmail-smtpd program just handles the SMTP conversation on a
socket. To be clear, it will not setup the socket for listening or wait
for connections. It relies on some other program to do those things. This
is an interesting design choice because it allows us to insert other
programs to perform checks in front of qmail-smtpd.

In the past, most servers on Unix would use the
inetddaemon to setup and listen on sockets. qmail, being a
program from that era, was probably just conforming to that norm. These
days, it is rare for a server to use an inetd since it is
poor at handling lots of connections or a hostile Internet. The
interesting thing here is that the older design style allows us to add new
filtering capabilities to qmail without having to change any of the
existing qmail code. So in order to solve the inetd
deficiencies and to keep the system extensible, DJB wrote another small
program called tcpserver. Its sole purpose is to perform
socket setup, filtering, and listening. It has several parameters for
setting TCP options, checking a simple list of IP addresses to block, or
performing anti-spoofing checks. When a connection does arrive,
tcpserver sets some environment variables and starts the
program specified on it's command line. The main environment variable we
are concerned with is TCPREMOTEIP,which contains a string
representation of the remote host's IP address.

Normally, in a qmail installation, tcpserver will run
qmail-smtpd as its next program in the pipeline. However, we
want to filter our incoming SMTP connections before we accept an
email. Therefore we need something to query our Trustic account via IP4R
before we accept an email via TCP. As luck would have it, DJB wrote a
program to do just this. That program is called rblsmtpd.

The rblsmptd program, as you'd expect, does one basic
thing. It checks the incoming connection using IP4R. If all goes well, it
runs a program specified on its command line. This is that same basic
pipeline API of handing off the socket. In order for rblsmtpd
to do its job, it checks the TCPREMOTEIP environment
variable, making an IP4R query against some system specified on the
command line. Depending on the outcome of that and the influence of a few
command line arguments, it will either end the SMTP transaction or run the
next program. In our case the next program to run will be the
qmail-smtpd.

rblsmtpd has some important command line arguments that
have important semantics, so I'll cover them in detail here. (These
descriptions are straight from the documentation). They handle how your
system deals with an IP4R request success or failure. That success or
failure in turn determines how and when mail will be rejected. Obviously,
this is important.

Command line options for rblsmtpd:

-r base: use base as an IP4R/RBL source. An IP
address a.b.c.d is listed by that source if d.c.b.a.base has a TXT record.
rblsmtpd uses the contents of the TXT record as an error message
for the client.

-B: (default) use a 451 error code for IP addresses listed in
the RBL.

-b: use a 553 error code for IP addresses listed in the
RBL.

-C: (default) handle RBL lookups in a "fail-open" mode. If an
RBL lookup fails temporarily, assume that the address is not listed.

-c: handle RBL lookups in a "fail-closed" mode. If an RBL
lookup fails temporarily, assume that the address is listed (but use a 451
error code even with -b.

For my system, I chose the following change to my tcpserver's
startup line:

Primer on Mail Bounces

In order to understand the effects of these, let met give a quick
primer on mail rejection or bouncing. There are really only two forms of
mail bounces, in layman terms, "Hard Bounce" and "Soft Bounce". A Hard
Bounce causes the sending mail system to give up immediately on delivering
an email, generating an error email or "bounce" to be delivered to the
sender. A Soft Bounce causes the sending mail system to give up
temporarily on delivering an email. The sending mail server may then try
again after an hour or some other specified timeout. If the mail
continues to get a "Soft Bounce" code from the receiving mail server for
some specified time (between two days and a week), the sending mail server
will give up and deliver a bounce message. In both cases, the sending mail
server takes responsibility for generating or delivering the bounce.

In the above configuration, I chose the -b and
-c command line parameters. With -b, I'm
choosing to cause a Hard Bounce to any email from an untrusted host. This
is important because I want senders to know immediately if there is a
problem. With the -c parameter, I'm choosing to Soft Bounce
if Trustic should happen to have any downtime. I wouldn't want to allow
spam into my system just because of a small outage and I don't mind
delaying that email. Besides, I monitor my systems so I could change this
if it ever happened for a long time. If someone were blocked, they would
know immediately that the email delivery failed.

What Would a Blocked Sender See?

Finally, Trustic provides the optional TXT record that the IP4R
protocol specifies (see previous article on the IP4R protocol). The
rblsmtpd program will include that message in the SMTP error
code. The sending mail servers will then copy that SMTP error into the
bounce message to the sender. Trustic puts a URL into the message so the
sender can go to a site and read the details on why the block
occurred.

As always, test your setup once you have it set up. You wouldn't want
to lose mail for half a day because you didn't test something.

Flipping the Switch

After I set the system up, not much happened. I sent recommendations
and only some of them blocked spam. The service was too new to be
effective. However, after about fifty users joined, things started to
happen. Soon I was blocking about 8 to 10 spam a day without making those
recommendations. After a few weeks, the real test came. I got hit with
1500 email attempts to my server. The attack wasn't as severe as the
previous one, but it could have caused the same problems. This time,
however, it produced zero load on my system. I was dropping all of the
requests. I didn't even know it was occurring until I checked my logs for
that day. Success!

After that, another interesting thing happened. After a while, my
recommendations were having a good effect on the other mail systems using
Trustic. I could see in the reports that some of my recommendations were
blocking hundreds of spam emails from being delivered to other systems. It
felt really good to give back to the others that helped block spam for
me.

Conclusion

I hope these two articles have been achieved their goal of providing
some good coverage of network level spam defenses. From my own recent
experiences, I have seen how the use of some simple, existing protocols
and a trust network could become a serious deterrent to the new spam
attacks. In the future, these defenses may protect us from more than just
spam. I look forward to seeing more people joining in and applying these
systems to their own networks.

Resources

Dru Nelson
has been on the Internet since 1988. After starting an ISP in
Florida, he moved to the San Francisco Bay area and has been involved
with large Internet infrastructure at companies like Four11 (Yahoo
Mail), eGroups (Yahoo Groups), and Plaxo. He is now the
CTO and co-founder of BrightRoll.com.