Greylisting with PF

This article will show you how I am using PF (the Packet Filter from the OpenBSD project) and spamd (also from OpenBSD) to implement greylisting and greatly reduce incoming spam. This solution is completely MTA agnostic. You do not have to make any changes to your existing mail server configuration to use spamd/PF. In fact, you can easily use PF and spamd to guard any SMTP mail server; even MS Exchange.

I first read about PF when reading Micheal Lucas's Absolute OpenBSD. If you have never read one of his books, I urge you to do so. His writing style is very clear and easy to follow. The book contains about thirty pages on PF, yet there is so much packed into that short chapter that when I read it I knew one day I'd start using PF. That day came a few weeks ago. My gateway at home has been happily running PF since early October. And more importantly, I've been happy. One of the best features of any software product is simplicity of use. PF is simple to start and easy to extend when you need the more advanced features.

On a related note, I'll also give you a short introduction to OS fingerprinting and how you can use PF to block/pass packets based on the sending OS.

How Greylisting Works

spamd and PF work together. PF will direct any known good clients directly to the mail server. Similarly, it will send known bad clients to the tarpit, a slow-responding mail server that does not add much load to your system. Finally, it asks new clients, not known to be good or bad, to try again later.

This makes three lists of clients:

Whitelist: known good clients.

Blacklist: known bad clients.

Greylist: we don't know if they are good or bad yet, but we will soon decide.

The key to greylisting is knowing that good and kind mail servers do not mind being politely asked to come back later. This is part of the STMP protocol. Spammers, however, are cheap and nasty. They don't like this. They probably won't come back later. Why? Queue management is tricky. If you're sending to millions of email addresses, why worry about a few messages that you cannot deliver? Just skip along to the next one. Spammers work on volume. Greylisting exploits that characteristic.

Implementing PF

There are several ways to enable PF, including compiling it into the kernel, or loading the module. I added a few options to my /etc/rc.conf file to ensure PF starts up at boot time. It also starts the logging daemon.

pf_enable="YES"
pflog_enable="YES"
pf_rules="/etc/pf.rules"

Note that I have chosen a nondefault value for my PF rules. The default value, as found in /etc/default/rc.conf, is /etc/pf.conf. To avoid any merge conflicts with mergemaster(8), I chose a different filename. The default install comes with many fine examples in /etc/pf.conf and I urge you to read them.

There are no filter rules loaded. To test your rules, without loading them:

pfctl -n -f /etc/pf.rules

This will report any syntax errors. To load the rules:

pfctl -f /etc/pf.rules

Read the man page for more -s options. You can pull out NAT, RDR, filtering, etc.

Very Simple Rules

This section shows very simple rules. It is a trimmed down version of what I use at home. fxp0 faces my ISP. fxp1 talks to my home network. This setting in /etc/rc.conf allows the gateway to forward packets between the two NICs:

3-4 are macros that define my internal network and my public IP address.

8 is my development web server.

9-13 are machines that are allowed to SSH to my gateway.

13 is a table of those machines.

17 is my nat rules.

18 allows redirection of port 80 on my gateway to my web server.

30 allows those machines to SSH in.

I hope that's enough to get you started with your own rule set.

Treating Different OSes Differently

Equality is a nice concept. The theory is great. The practice is not universal, especially when it comes to operating systems. Why discriminate? I will answer that question in the form of a story related to me by someone who uses OS fingerprinting at home to lessen his domestic workload. To protect the guilty, I will call this person Phil.

Phil uses BSD in his everyday work. At home, his kids use BSD. Their machines also dual-boot with Windows in case they need to do specific things for homework. In general, they use BSD. When using Windows, they can only use the Web; nothing else. Why do these rules exist? To lessen Phil's workload. He doesn't want to remove viruses and spamware all the time. How do you keep them off the Windows computers? You keep the Windows computers off the Internet.

Sure, rules are nice, but some kids, and some adults, don't follow the rules. It would be nice if there was some way to enforce this at the firewall. Enter OS fingerprinting.

OS fingerprinting is not new. nmap, for example, uses it. From man pf.conf:

Passive OS Fingerprinting is a mechanism to inspect nuances of a TCP connection's initial SYN packet and guess at the host's operating system.

PF's ability to detect the Operating System at the other end of the TCP/IP connection allows Phil to prevent his kids from breaking the rules. To illustrate this, I'll try blocking incoming queries based upon the OS you use. On my web server at home, I created a website: http://beta.freebsddiary.org/. To demonstrate that the website is actually running, you will be able to browse freely to the URL, but if you try http://beta.freebsddiary.org/:8080, you will not be able to connect if you use Windows.

How do I do this? I'll show you the PF rules, but the Apache setup is outside the scope of this article.

Redirect Incoming Traffic to the Web Server

These rules redirect incoming traffic from the gateway to the web server:

rdr on $ext_if proto tcp from any to $external_addr port http -> $webserver
rdr on $ext_if proto tcp from any to $external_addr port 8080 -> $webserver

As you can see, port 80 (http) and port 8080 both redirect to my web server.

Block/Pass Traffic on Those Ports

These rules will pass or block the traffic based on port and OS:

pass in quick on $ext_if inet proto tcp from any to $webserver port http flags S/SA synproxy state
block in quick on $ext_if inet proto tcp from any os windows to $webserver port 8080
pass in quick on $ext_if inet proto tcp from any to $webserver port 8080 flags S/SA synproxy state

The first line allows traffic to flow freely from my internal NIC to the web server on port 80.

The second line blocks all traffic from any Windows machine headed towards port 8080 on my web server.

The last line passes all traffic on port 8080. The above line contains a quick directive so if the client OS is Windows, subsequent filter rules have no effect on the packet.

I originally wanted to redirect different OS connections to different web servers, but the OS directive is not available on the RDR statement.

Enabling spamd

For FreeBSD, spamd comes as a port. The easiest way to install it is to have a fresh copy of the FreeBSD ports tree and issue the commands:

cd /usr/ports/mail/spamd
make install clean

To enable spamd, get greylisting going, get verbose logging, and add these entries to /etc/rc.conf:

pfspamd_enable="YES"
pfspamd_flags="-g -v"

See man spamd for more details on the various options you can specify.

If you are using greylisting, also run the command:

mount -t fdescfs fdescfs /dev/fd

This mount allows spamlogd to update the spamd table. To mount this directory at boot time, add a line to /etc/fstab:

fdescfs /dev/fd fdescfs rw 0 0

To ensure you have the latest versions of the spam blacklists, refresh them once per hour with a line in /etc/crontab:

48 * * * * /usr/local/sbin/spamd-setup

The spamd-setup utility adds blacklists by adding addresses to the PF table <spamd> according to the instructions in /usr/local/etc/spamd.conf. To distribute the load a bit and avoid having everyone hit the servers at the same time (perhaps 48 minutes past the hour, or at the top of the hour), change 48 to whatever minute it is when you enter the crontab entry.

Make a copy of spamd.conf:

cp /usr/local/etc/spamd.conf.sample /usr/local/etc/spamd.conf

You may wish to amend spamd.conf according to your needs. My changes are:

all:\
:spamhaus:china:korea:

to:

all:\
:spamhaus:spews1

I also added a line to /etc/syslog.conf so I could see the log from spamd:

!spamd
daemon.err;daemon.warn;daemon.info /var/log/spamd

I created the file:

touch /var/log/spamd

Remember to HUP syslogd so it reads your changes and takes appropriate action:

kill -HUP `cat /var/run/syslog.pid`

Although your logfile will be empty at this point, here are a few entries that appeared after mine had run for a while.

No, those aren't the real email addresses from my logs, but they are close.

Known Good Mailers that Have Trouble with Greylisting

There are some problems with greylisting. Be sure to read the whitelisting section at greylisting.org. I have taken its whitelist and added it to my whitelist (/usr/local/etc/spamd-mywhite). Also in that file are my own mailservers and any special places which are immune to any spamd intervention.

3 declares a table of locations to exempt from greylisting. I maintain this table manually.

6 makes everyone in the whitelist go straight to the mail server.

7 makes everyone on the greylist go straight to spamd.

8 forces everyone not on my whitelist to talk to spamd.

10-11 feed data to pflog, which spamlogd will monitor and use to update the spamd table.

Note that the use of pass on the RDR rules is significant. Here's an explanation from the OpenBSD PF FAQ:

NAT and Packet Filtering

NOTE: Translated packets must still pass through the filter engine and will be blocked or passed based on the filter rules that have been defined. The only exception to this rule is when the pass keyword is used within the nat rule. This will cause the NATed packets to pass right through the filtering engine.

Also be aware that since translation occurs before filtering, the filter engine will see the translated packet with the translated IP address and port as outlined in How NAT Works.

In short, if you're on a whitelist (either spamd's whitelist or my whitelist), you go straight to the mail server. Everyone else goes to spamd.

On your first visit to spamd, you are asked to come back later. If you do, then you're asked to try again and are added to the whitelist.

By the way, after making changes to /usr/local/etc/spamd-mywhite, tell PF to notice the changes:

I have removed some of the columns from this display to make it fit better on the page.

On a side note, I'd like to see spamd running as something other than nobody. Perhaps I'll work on that later.

spamd-setup maintains the <spamd> table shown on line 1 of the PF rule set found later in this article. To view the contents of this table:

pfctl -t spamd -T show

From the Greylist into the Whitelist

You don't have to worry about moving items from the greylist to the whitelist. spamlogd will take care of that for you. If you're setting this up for the first time, start spamlogd:

/usr/local/libexec/spamlogd

To ensure that spamlogd starts at boot, add a line to
/etc/rc.conf:

pfspamlogd_enable="YES"

spamlogd updates the spamd database (/var/db/spamd). When it sees a successful connection, spamd in turn uses this database to decide whether someone is on the whitelist or greylist. To provide spamlogd with the information it needs, you must log your mail server activity. See lines 13 and 14 in my example PF rules. Read all the details in man spamlogd.

If spamlogd does not start, it is probably because pflogd is not running. Start it. This is how spamlogd looks when it is running:

A Sample Greylisting

It's helpful to send a message from a non-whitelisted server and demonstrate how the server moves from the greylist to the whitelist. I will be sending from dan@zip.example.org to dan@nyi.example.org. For your information, zip runs Sendmail and nyi runs Postfix. In both cases, that is completely irrelevant to greylisting.

Here is an extract from the sending mailserver. I guess I should point out that this server is in New Zealand and the one I'm sending to is in New York.

There it was, clear as day. The entry has been greylisted. The three numeric fields indicate timestamps related to this host. The 2 means the host has attempted delivery twice. The 0 means the host has not yet delivered any mail.

Why was the second attempt not allowed? spamd has three time parameters related to greylisting. See the man page for better definitions. The values shown are the defaults.

passtime: if after this time period, spamlogd sees a retried delivery, it will move the server to the whitelist (25 minutes).

greyexp: entries on the greylist will be removed if there have been no retries within this period (4 hours).

whiteexp: entries on the whitelist are removed if there has been no mail activity in this time period (36 days).

The default passtime value (see man spamd) is 25 minutes. A host will remain greylisted for at least 25 minutes before it can move to the whitelist. What will move it to the whitelist? A retry after passtime minutes. This requires three delivery attempts; the third will succeed if it occurs after the passtime period and before the greyexp period terminates. By default, the sending mailserver will be greylisted for 25 minutes, and then has until four hours after the first delivery attempt to try again. After the greylisting period expires, the sending host must go through the greylisting process again.

There you go, the entry has been whitelisted. Three delivery attempts have occurred, and none have succeeded. Actually, as I type this, the mail is still sitting on zip, waiting to attempt another delivery. This time, PF will redirect the incoming connection directly to my mail server, not to spamd.

203.118.144.46 will remain on the whitelist until 36 days of no sent email. While on the whitelist, it will not be subject to greylisting. If if falls off the whitelist, it will go through the above greylisting process again.

Problems I Encountered

Yes, I had a problem. The <spamd-white> table was always empty, even after spamlogd moved something from the greylist to the whitelist. My thread whitelists clients still being greylisted in the FreeBSD PF mailing list found the missing piece for me:

Now I had an even bigger problem. Despite having the whitelist updated, my email still wasn't getting through. Whitelisted clients were still subject to greylisting. Delo found the answer. My rules were wrong. I had:

See the problem? Line 1. That table name is wrong. It should be <spamd-white>, the table maintained by spamlogd. D'oh! When I found this problem, I corrected /etc/pf.rules and updated PF by issuing the command:

pfctl -f /etc/pf.rules

Later on, I found out about this option:

-N Load only the NAT rules present in the rule file. Other rules
and options are ignored.

Once I fixed the NAT rules, I went to m21 and tried to connect. I made it straight through to the real SMTP server:

Good, that proves the whitelisting is working. Then I flushed the Postfix mail queue, and the mail message went straight through.

Yes, I missed this entirely during the port install:

$ cd /usr/ports/mail/spamd
$ less pkg-message
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
In order to use spamd greylisting feature you have to have a mounted fdescfs(5)
at /dev/fd. This is done by adding:
fdescfs /dev/fd fdescfs rw 0 0
to /etc/fstab. You may need either a customized kernel, or kldload the fdescfs
kernel module.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
$

Yes, I have slightly obscured the domain names, but you should be able to see who is sending to what. For the record, the MX server in question is not an MX for langille.org or freebsddiary.org... but that's not stopping the spammers from trying. At present, only bsdcan.org uses this greylisting server as an MX. I'm about to add more domains to it and implement greylisting on my other servers.

As I type this additional note on November 24, about 3 weeks after the above, here are the stats of each of my three mail servers:

nyi

$ spamdb | grep -c GREY
101
$ spamdb | grep -c WHITE
4462

havoc

$ spamdb | grep -c GREY
256
$ spamdb | grep -c WHITE
2404

supernews

$ spamdb | grep -c GREY
30
$ spamdb | grep -c WHITE
37

It is interesting to see that one machine has whitelisted nearly 4500 servers in about nine days.

Greytrapping

I'm sure all of this sounds great. It can be better. Greytrapping is one step further than greylisting. No doubt you have an abandoned email address that still receives mail. It's probably been on spamming lists for years. If someone is sending email to that address, it's bound to be spam. You can add that address to spamdb as a spamtrap address. See man spamdb for details. For example, to designate anyone sending to yourname@example.org, use the command:

spamdb -T -a "<yourname@example.org>"

I have a list of 24,592 such email addresses. Why? Well, they aren't really addresses. They are Message-ID values from FreshPorts. FreshPorts didn't always store Message-ID. When I added that attribute, I needed to come up with a value for the existing commits stored in the database. Unfortunately, I selected something like fp1.12345@example.org (s/example/FreshPorts/). Spammers grabbed all those addresses, and I started to see huge spam attempts. All bounced of course, because they were not valid addresses. I have since changed those Message-IDs to @dev.null.example.org (s/example/FreshPorts/), but the spammers continue.

So how do I get the email addresses into spamdb? They are all in a file named greytrap. This command loads them. It takes a few minutes to complete.

cat greytrap | xargs -n1 spamdb -T -a

That's all there is to it.

Greyscanning

With newer versions of spamd (not available in the FreeBSD Ports tree at the time of writing), you can take advantage of the greylisting period to scan your logs and take appropriate action. The greyscanner script will scan the spamdb output and look for patterns and blacklist those IP address for 24 hours. If it's not spam, it will come through later. If it is spam, well, you've delayed it. This script can validate the address, check for an MX or A record for the source address, and more.

Things to Think About

Greylisting can delay mail. Greylisting can block mail, but only if you continuously redirect the connection to the tarpit. However, it does greatly reduce the amount of incoming spam. I have no comparative statistics to show you. All I know is that I like it and that it reduces the amount of garbage in my mailbox. :)

Dan Langille
runs a consulting group in Ottawa, Canada, and lives in a house ruled by felines.