Mail server

After having depended on my friends for a few years, for web and email hosting, I recently acquired a private server again. In this HOWTO I will try and document the entire mail setup.
At the end of this article, I will have a mail server that does:

Postfix MTA (Mail Transfer Agent)

Dovecot IMAP (Internet Message Access Protocol)

Let’s Encrypt (SSL certificates)

Dovecot SASL (Simple Authentication and Security Layer)

mail accounts (partly) separated from system accounts

Dovecot LDA (Local Delivery Agent)

Sieve / ManageSieve

Greylisting

RBLs (Real-time Blackhole Lists)

Bogofilter (spam filter)

ClamAV (virus scanner)

SPF (Sender Policy Framework)

DKIM (DomainKeys Identified Mail)

Just so you know that there’s more to mail than just typing a text and hitting the Send button…

Currently my server only serves 1 user with 3 domains. But with the setup I describe here, it is extremely simple to upgrade that to many users running many domains.

This tutorial is not for the complete layman, but should be comprehensible for those who have played around with Linux and its services and command line a bit. I have tried to make it interesting and useful for both the beginning and the experienced sysadmin.

Corrections, clarifications, supplements and typos are welcomed at mailtuto@ohreally.nl.
Spam is not, as you will understand after reading this article.

License

What that means, in short, is that you may copy, adapt and/or redistribute this text and the scripts and examples contained in it, but only if you give appropriate credit to the original author (me), and share your own contributions under the same or a compatible license.

To find out how to give appropriate credit, to find information about compatible licenses, or to read the full license text, follow this link.

This license does not, in any way, limit your rights to link to this article at www.ohreally.nl.

Warranty and responsibility

Before I discuss anything else, it is important that you know and understand that there is no warranty, and that I do not accept any responsibility for anything you do or omit on the basis of this documentation.

The below works for me, but that does not automatically mean it will work for you as well.

If you do decide to copy any or all of the actions I describe here, I strongly advise you to create backups where needed, and not to experiment on critical production servers.

About this document

Well, it’s a long read… Do not count on setting up your mail server in 1 or 2 hours.
Most other websites would split this tutorial up into 15-20 pages. But since my website doesn’t have any banners that are paid per view, I don’t have any reason to make you go back and forth. With the advantage that you can now use Ctrl-F to search through the entire tutorial, instead of just a chapter.

The setup I describe is suited for a small-ish mail server; think company or organisation (or family/friends). I think that once you pass 100-150 users, you may wish to make some changes; for example disconnect mail users from system users entirely.
But even for larger, or even much larger setups, there should be some valuable info below.

All files mentioned in this document should be created and/or edited as the root user, unless stated otherwise.

For commands, a root# prefix indicates that the command should be executed as root user, and a userx$ prefix indicates that the command should be executed as regular user. For example:

userx$ sudo bash
root# ls /root/

If you’re inclined to include the root# or userx$ prefixes in the commands, you’re probably not ready for this tutorial, and should learn some Linux basics first.

This article was published on 19 November 2018. If you’re reading this after, say, 2023, you may want to bookmark it, and see if you can find something more recent first.

Background and preparation

Store your spam

This is the first step in your preparation.
You probably receive mail somewhere already. Configure your mail account and/or client so that no spam is thrown away. Instead put it all into a separate folder. We’ll be using this later to train the spam filter.

Operating system

The operating system I’ve chosen for my server is Gentoo Linux. For those wondering why, I’ll very briefly indicate my motivations.

I had 3 requirements:

because of support for software I’d like to experiment with later, I needed it to run Linux and not *BSD, which I’d normally prefer for a server

it had to be a rolling distribution

it may not use systemd

and I had 1 wish:

I would appreciate a software installation system like the ports collection on *BSD

Gentoo was the only OS that met my 3 requirements, and as a bonus met my 1 wish as well. So it wasn’t a difficult choice.
Even though I have 20-25 years of experience running and administering Linux and *BSD servers and desktops, this is the first time I use Gentoo.

Software

For the setup that I envision, all the software listed below needs to be installed. If, on your current system, software is installed that overlaps with the software we need (e.g. Sendmail is installed, while I say we need Postfix), I leave it up to you and your imagination to decide how to deal with that; obviously, it would be appreciated if you could document your choice and/or solution somewhere, for people who run into that same problem after you have solved it.

To fine tune software installations, Gentoo uses so-called USE flags. Below I describe the USE flags for the software I installed; if you’re on a system other than Gentoo, you will have to find out for yourself whether and how you need to fine tune the installation (and again: please document).
The USE flags you need may be stored in /etc/portage/package.use/*, in a simple text file per application.

mail-mta/postfix

The MTA (Mail Transfer Agent) is the software that receives the mails that are delivered to the server, and handles them further. This concerns the mails that are sent from elsewhere on the internet to me, but also the mails that I send to addresses elsewhere on the internet. The handling may consist of the invocation of a spam filter or virus scanner on the mail (for incoming mail) or, for example, authenticating and authorising me to use this server to send mail out on the internet (for outgoing mail).
The protocol handled by the MTA is called SMTP (Simple Mail Transfer Protocol) or SMTPS (the SSL encrypted version of SMTP).

net-mail/dovecot

There are 2 protocols that allow me to open the mails received by my server, on my pc or smartphone: IMAP (Internet Message Access Protocol) and POP3 (Post Office Protocol, version 3); sometimes also refered to as IMAPS and POP3S, the SSL encrypted versions of those protocols. Dovecot is an application that can present my mails to me, using those 2 protocols. I choose to only use IMAP, though; POP3 moves the mail from the server to the computer, while IMAP leaves the mail on the server, permitting one to reach it from multiple computers.

I chose Dovecot, because I have been a satisfied Dovecot user for a long, long time, and I don’t really know any other IMAP servers. Apart from serving messages over IMAP and POP3, Dovecot also does:

Dovecot SASL (Simple Authentication and Security Layer)
asks for my username and password when I want to read my mail, but can also be used by Postfix to ask for my username and password when I want to send mail (see /etc/portage/package.use/mail_mta-postfix above)

app-admin/sudo

Chances are, Sudo is already installed on your system. And if it isn’t, it should, and not only for this mail setup.

In the setup we’re doing now, sudo will be used to allow users to change their email passwords.

app-admin/sudo does not need any special USE flags for this mail server setup.

app-crypt/certbot-apache

Let’s Encrypt is an organisation that gives out free SSL certificates. We will be using those to encrypt the communication between the pc or smartphone, and the server. Certbot is an application that may be used to request, retrieve and install those certificates.

I chose certbot-apache, because I have the Apache web server installed. If Nginx is your web server of choice, then install certbot-nginx. And at the Let’s Encrypt website you’ll find even more solutions.
Below, I will only briefly glance at the retrieval of SSL certificates, to give you an idea of how it works. But I will not elaborate on it, because I don’t want to extend this text, which is very long already, with the installation and configuration of a web server. You run Linux, so I expect you to be able to configure a web server, or at least be able to find the information you need to do so.

app-crypt/certbot-apache does not need any special USE flags for this mail server setup.

mail-filter/postgrey

Emails travel from mail server to mail server: the sender delivers the mail to his mail server (or his provider’s mail server, or Gmail’s), and that mail server delivers the mail to the recipient’s mail server; there may even be more servers between those. This is the Simple Mail Transfer Protocol that I described above.
When, on delivery, the receiving mail server returns an error, the sending mail server will retry later. Spamming software, however, is often developed to discard any mails that are undeliverable for some reason; a retry for each undeliverable mail would take too much time.

Greylisting (between blacklisting or ‘reject’, and whitelisting or ‘accept’) is a technique that utilises this to fight spam: each incoming mail is rejected with a temporary failure, the first time it is presented. A legitimate, well-configured mail server will retry later, while spamming software gives up. The second time a same mail is presented, it is let through.
I don’t have exact figures, but the logs on my server tell me that a lot of spam presented to my server is successfully dropped using this technique. So these mails never reach my spam filter, which saves a whole lot of CPU cycles and memory, which may now be used for other things.

mail-filter/bogofilter

My spam filter of choice is Bogofilter. The only other project that is not dead or dying, or under-developed seems to be Spamassassin, which I find way to heavy. And since I don’t need all it’s bells and whistles, I can save myself some CPU cycles and memory, and install Bogofilter.

Bogofilter is a learning, so-called Bayesian, spam filter. This means that it learns from the mails that are classified by the user as being spam or non-spam, so filtering improves with time, as Bogofilter adjusts itself more and more to your personal mailbox.

mail-filter/bogofilter does not need any special USE flags for this mail server setup.

app-antivirus/clamav

As I’ve been using Linux/*BSD exclusively for a really long time, I have never really worried about viruses or virus scanners. But because I’m documenting this setup for others (you!), who may be using Windows on their home pcs, I decided to add a virus scanner for completeness.

This is /etc/portage/package.use/app_antivirus-clamav:

# Enable the milter interface.
app-antivirus/clamav milter

mail-filter/pypolicyd-spf

You’ve probably received spam sometimes, that had your own email address as the sender address. This is because one can put any sender address one wants on an email.

SPF (Sender Policy Framework) is a protocol that tries to fight this by enabling sysadmins to indicate in the DNS which mail server(s) may be used to send mail for a certain domain. The receiving mail server may then verify with a simple DNS request whether a mail for a certain sender originates from the correct mail server. If this is not the case, the sender address is likely to be forged, and the mail is likely to contain spam or a virus.

mail-filter/pypolicyd-spf does not need any special USE flags for this mail server setup.

mail-filter/opendkim

DKIM (DomainKeys Identified Mail) is in part a bit comparable to SPF. The sending mail server adds a signature to the headers of the mail, which may be verified with a DNS request to the sender’s domain’s DNS server. Furthermore, this signature may be used to verify that the mail has not been tempered with between the originating server and the receiving server.

mail-filter/opendkim does not need any special USE flags for this mail server setup.

net-dns/bind

The ISC Bind DNS server. Where DNS stands for Domain Name Service.
I would have prefered to use djbdns, because of it’s extremely simple configuration, but that project seems to have passed on.

To tell the server for which domains it handles the mail, we do not need a DNS server; we will specify that in the MTA configuration. But we do need DNS to tell other mail servers where to deliver the mail for our domains. And to host the DNS TXT records for SPF and DKIM.

net-dns/bind does not need any special USE flags for this mail server setup.

(You can put all that on a single line. If you break it up into multiple lines, as I have, each line ends in backslash-enter, with no space in between).

More…

System account(s)

As said, my server currently only serves 1 user. In this documentation, that user will be called userx. However, this setup is easily extendable to more users, so pick as many names as you want.
I will not document the creation of users; if you don’t know how to create users on a Linux box, maybe you’re not quite ready to set up a mail server.

The user needs to have a home directory, to save Sieve scripts to.
Currently, the user also needs shell access, to be able to set/change their mail password. I’d advise you to only use SSH keys for shell access, and disable password authentication; this way the password can never be cracked (or copied in an internet cafe, or anything).

Someday I hope to create a web and/or email interface for changing the password. If someone beats me to it, though, I promise I won’t get mad; send me a link to the source and documentation, so I can link to it.

Domain name(s)

As said as well, I currently have 3 domains. In this documentation those will be example.com, example.net and example.org.
The fully qualified hostname for the server we’re setting up will be yellow.example.com (no particular reason; most sysadmins have a theme for their hostnames, so why not colours?). We’ll get to the other names for this server when I talk about configuring the DNS.

It’s not mandatory to use 3 domains, but you do need at least 1. If you set up this mail server to play with on your own local network, pick a domain name that ends in ‘.local’, to make sure that you’ll never clash with domain names on the internet.

If the server is connected to the internet, and using officially registered domain names, it is up to you to make sure that the chosen domain names, or at least their MX records and the hostname, resolve to this server.

Directories

Sometimes mail is served from the user’s home directory. I have chosen not to do so, but instead use 1 single central directory whence all mail for all users is served. This eases backup, for example.

root# mkdir -p -m 751 /service/mail

And certbot-apache is going to need a virtual host to verify the server’s address for the SSL certificates. My websites are also served from a single central directory.

root# mkdir -p -m 751 /service/www

Configuration

The basics

The basics, for me, would be everything we need to be able to send and receive email, that is working SMTP and IMAP servers, including user authentication and SSL encryption for the communication between pc and server. Which is quite some work, already.

As said before, I don’t use POP3. If you want to support POP3, I suggest you follow this tutorial first, and then use the knowledge you’ve gained to add POP3 support.

Firewall / packet filter

Before anything else, let’s pierce some holes in the firewall; typically something one forgets when debugging problems.
Obviously, if you’re not running a firewall on your server (even though you should!), you skip this step.

There are many firewalls, packet filters and whatever they’re called, so I can’t give you the details for yours; maybe I’ll do a little article about packet filtering some other time.
Anyway, for this setup, you’ll need to open up these incoming ports on the server:

25 (smtp ⇒ Postfix)

53 (dns ⇒ Bind)

443 (https ⇒ Apache, for Certbot)

587 (submission ⇒ Postfix)

993 (imaps ⇒ Dovecot)

4190 (sieve ⇒ Dovecot)

DNS

I will assume that you will have domain name registrar (not yet, later) delegate your entire domain to this server (this server will be the authoritive primary DNS server for your domains). If, for some reason, you decided to do it differently, I guess you know Linux/DNS well enough to be able to improvise.
And if you’re not using an officially registered domain name, remember to use a domainname that ends in ‘.local’; that could be just domainname local with hostname yellow.local, or domainname mycooldomain.local with hostname yellow.mycooldomain.local.

We’ll pretend that our assigned IP address is 198.51.100.157; you should, obviously, replace that with the actual IP address for your server.

So, as said before, for this document we will be serving the domains example.com, example.net and example.org, and our server is known as yellow.example.com. We could make the MX record for our domains point directly at this hostname, but let’s follow conventions and create a separate name for MX; we’ll just call it mail.example.com, and it needs to be an A record, because MX records may not point at CNAMEs. And then, just because it looks pretty in our email client configuration, we will create 2 CNAMEs indicating the protocols we serve: smtp.example.com and imap.example.com.
Finally, for completeness, we will add ns.example.com to refer to the name server, and www.example.com to host a website; the first must also be an A record, the latter may be a CNAME.

And if you’ve paid attention, you’ve seen that we should not forget to create an email alias sysadmin@example.com when we get to the email aliases.

One thing to remember: each time you change these files, increment the Serial. If you don’t, you’re changes will not be propagated to the internet.

Usually there is a second NS record, pointing to the secondary name server, which could be your colocation provider, a friend’s server, or a free or paid service elsewhere on the internet. So, go find yourself a secondary, and add it below the first NS record. (You can do this later if you want, but just don’t forget it; it’s important, but not critical.)
Obviously, some configuration will need to be done on their end, and you will have to allow transfer requests from their server; the service you select will surely have documentation on this.

If you’ve just installed Bind, this is the moment where you add it to the default runlevel and start it:

root# rc-update add named default
root# rc-service named start

On the other hand, if Bind was installed and running already, you only need to reload the configuration:

root# rc-service named reload

And this is the moment where you contact your domain name registrar to have them delegate your domain to your server.

Now, there is one ‘but’: what if you already have mail for these domains running at some other server, and you don’t want to switch servers before this server is all ready to process incoming mails?
Well, in that case, have the MX record point at the other server for now (don’t forget to increment the Serial, and reload named) before you contact your registrar. That way, your mail will still be delivered at the other server, but at this server we can continue the new setup.

SSL certificates

Now that DNS is up and running, we can get our SSL certificates.
Or actually, we only need a certificate for the example.com domain, because all the MX records point to mail.example.com, and for reading and sending mail we will use imap.example.com and smtp.example.com respectively.

To get our certificate from Let’s Encrypt, we use certbot. This is how it works, roughly:

we create a virtual HTTP host

certbot places a file in the virtual host

certbot sends a request to Let’s Encrypt

Let’s Encrypt does a HTTP request to get the file created in step 2

if the file is found, we have proven that we own this domain, and we receive the certificate

Apache web server configuration is really beyond the scope of this tutorial, so I will just give you the configuration for the virtual host.

and see if you can reach mail.example.com in a web browser.
But Apache will refuse to (re)start because it can’t find the SSL certificates. You could temporarily use one of the pre-generated certificates in /etc/ssl/. You can then change them back after the first certbot run. (Yes, I could have done it correctly right away. But are you here to copy stuff, or to learn stuff…?)

And then we can request the certificate, indicating the location of the VirtualHost document root, and the 3 ‘hosts’ for which it should be valid:

This will ask several questions. These questions will only be asked once; any next times you run certbot, your answers will be re-used.
It is important to remember or note down the email address you give, as you will have to create an email alias for this later. For the sake of this documentation, let’s say we picked sslcerts@example.com.

If your hostnames have previously been running on other IP addresses, it may take some time and some re-trying before the Let’s Encrypt server finds the correct IP address for verification.

If all has gone well, a directory /etc/letsencrypt will now have been created, holding certificates, keys and what not.
(If all has not gone well, you could try one of the other solutions listed on the Let’s Encrypt website.)

On some Linux distros, a cronjob for automatic renewal of the certificates is automatically installed when installing certbot, but not on Gentoo. Save this as /etc/cron.daily/letsencrypt:

#!/bin/sh
/usr/bin/certbot renew --quiet

Make sure the script is executable:

root# chmod 755 /etc/cron.daily/letsencrypt

This will check which certificates near expiry, and renew them, if any. This will work for all Let’s Encrypt certificates on the system, so you don’t need to create a second cronjob if and when you install other certificates for, say, a website.

Side note:
After creating the VirtualHost above, it won’t be long before your logs tell you that script kiddies have found it. Since certbot only needs the directory /service/www/mail.example.com/.well-known/acme-challenge/ (which translates to https://mail.example.com/.well-known/acme-challenge/), you may wish to put up a ‘catch all page’ for these 1337 h4x0r5. If you lack inspiration for the contents of such page, you could look up goatse or meatspin. 😉
This .htaccess will redirect all requests for non-existant files or directories to /skiddie.html:

Passwords

Both Postfix and Dovecot will need to authenticate users who wish to send or read mail. Actually, Dovecot will take care of the authentication for both, but we’ll get to that later. Let’s set up the password database first.

We’ve already created the system user userx. But Dovecot allows us to separate system passwords and mail passwords. And that’s cool, because now if ever a user has their mail password compromised, it won’t compromise the system account; someone who manages to copy a mail password of one of our mail users, will not automatically have shell access. We could even choose to not create a system password for our users, at all.

Allthough it’s tempting to go for a MariaDB password database, I decided to go for a plaintext database. Running a DBMS means more software to maintain, and another daemon that may crash. Maybe I’ll switch to MariaDB if and when I decide to create a web interface for setting the password. Or maybe not.

The mail passwords will be saved in the file /etc/passwd.mail, a file that doesn’t (or at least shouldn’t) exist yet. For now, all this file needs is the username and a colon:

userx:

The mail username should match the system username.

This is the script that changes the password; save it as /usr/local/bin/mailpass:

(Yes, you should do so now. If you don’t, you will forget it, and then later it will take you hours to find out why things don’t work.)

If your users do not have shell access, it should be quite trivial to create a simple web interface for this. If you do so, please open source it and notify me, so I can add a link here.

IMAP server

Well, finally we get to some real email stuff!

To start, I don’t want to do anything fancy, yet. We will just allow our user to read their mail, using the IMAP protocol, over an SSL encrypted connection. Well, actually that is quite fancy. But for now we won’t do any filtering and all that.

The configuration for Dovecot can be found in /etc/dovecot/ (what a surprise 😉 ). These are the changes I made for this first phase:

# /etc/dovecot/dovecot.conf
# Only serve IMAP.
protocols = imap

# /etc/dovecot/conf.d/10-auth.conf
# Do not disable plaintext authentication.
disable_plaintext_auth = no
# Do not include auth-system.conf.ext.
#!include auth-system.conf.ext
# Include auth-local.conf.ext.
!include auth-local.conf.ext

My version of Dovecot suffers from a bug where it requires DH parameters even when you’ve not enabled Diffie-Hellman key exchange. If your server also complains about that, add this line to /etc/dovecot/conf.d/10-ssl.conf:

ssl_dh = </etc/dovecot/dh.pem

And run this command to generate the file in question:

root# openssl dhparam 4096 > /etc/dovecot/dh.pem

This command will take a LONG time, especially on a new server, that hasn’t collected a lot of entropy yet. And I don’t mean Go grab a cup of coffee-long, but more like Go to work and hope it’s finished when you get back-long.

Your IMAP server should now be up and running. Try and connect to it using these settings:

server name: imap.example.com

protocol: IMAP

port: 993

connection security: SSL/TLS

authentication method: normal password (sometimes called plaintext)

user name: userx

password: you’ve set this above

And when you’ve managed to connect, it’s time for the final test.

Open a shell at the server as userx, and save this as ${HOME}/testmail:

If this mail arrives in the mailbox in your email client, then IMAP works (and LDA as well).

For more information on Dovecot’s configuration, head on over to the Dovecot wiki.

SMTP server

Okay, it’s nice to have a running IMAP server. But if you want a bit more than just sending yourself mails from the command line, we’re going to need an SMTP server.

Postfix listens for connections on 2 ports: 25 (smtp) where external mail servers deliver mails destined for us, and 587 (submission) where our authenticated users deliver their mails to have them sent out on the internet.(Just had this thought: would it be very nerdy to get a tattoo saying ‘587’ if you’re into BDSM? Okay, never mind…)
For our users to get authenticated, we’ll make use of Dovecot-SASL (Simple Authentication and Security Layer), which in short means that Dovecot will take care of the authentication, using the password database we created above.

The Postfix configuration, you guessed it, can be found in /etc/postfix/. These are the changes for this first phase:

The values for that last parameter look a bit funny, 3 times ‘permit’ in a row, but we’ll add some other values in between later.

I have the feeling that there is quite some overlap between my master.cf and my main.cf (my config is a collection of very, VERY many examples I found online). The Postfix documentation, at least to me, is not quite clear about the relation between the options with the same names, specified in the one file or in the other.
That said, my setup does what I expect it to do, so at least the options do not bite each other.

In the above configuration, we defined 2 alias databases: alias_database and virtual_alias_maps. These files define for which recipients the mail should be redirected to which other recipients.

/etc/mail/aliases

This file is for local aliases. The format is very simple:

original_recipient: new_recipient

If you open the file in a text editor, you will see that all (local) mail actually ends up with user root. So if you would send a mail to abuse@yellow.example.com, it would be redirected to postmaster@yellow.example.com, which would be redirected to root@yellow.example.com.

There is one thing to be added to this file: mail for root should be redirected to a user who actually reads their mail. Which in our case would be the only ‘real’ user we have: userx.
Add this line to the file, and add it somewhere near the top, so it’s easy to find:

root: userx

These days, with virtual domains, it’s quite rare to have to change more than that.

Since Postfix does not read this file directly, but prefers a database, which is more efficient, we generate that database by running the command

root# newaliases

/etc/postfix/virtual

The second aliases database defines aliases for virtual users. This database also consists of lines with an LHS (left-hand-side) and an RHS (right-hand-side). There are a few differences:

LHS does not end in a colon

LHS and RHS may contain a domain name

LHS defines a virtual user or virtual domain

RHS may indicate a local or virtual user

And for each virtual domain, there must be a line where the LHS defines the domain name, and the RHS can be any string.

accounting@example.com and sales@example.com will be redirected to office@example.com

office@example.com, including the mails redirected above, will be redirected to userx@example.com

userx@example.com will be delivered to local user userx

other addresses in the example.com domain will be bounced with a message indicating a ‘user unknown’ error

acquisition@example.net will be redirected to sales@example.com

sales@example.net will be redirected to sales@example.com

mail for any other address in the example.net domain will be delivered to local user userx

mail for any address in the example.org domain will be forwarded to that same address in the example.com domain, where it will be subject to the rules for that domain; so warehouse@example.org will be bounced, because warehouse@example.com does not exist

End of phase 1

That concludes ‘the basics’. You now have a fully functional mail server, and if you’ve had enough of this tutorial, you can stop here, and just enjoy what you have.
But if I were you, I’d continue to the next chapter…

If you do so, it might be a good idea to open an extra terminal and

root# tail -f /var/log/mail/current

for the rest of this manual.
Hit Ctrl-C to end that command.

Sieve

If you’re anything like me, you have about a zillion folders and subfolders in your mailbox. So, now that the server can receive mail, you may want to organize the automatic sorting of your incoming mails before doing anything else.
To this end, we’ve included Sieve support when installing Dovecot.

Sieve is a scripting language for filtering email messages. You could edit the scripts using your favorite text editor when you’re logged in at the server. But ManageSieve clients — and add-ons for email clients — exist as well, permitting the user to edit their filters in a more user friendly way, and using their SMTP/IMAP password.

The ManageSieve daemon listens on server port 4190. You connect with the same username and password as you use for SMTP and IMAP.
Personally I really enjoy the Sieve add-on for Thunderbird for editing my scripts; it has syntax check, a Sieve reference, et cetera. But on the other hand, I confess that I have not tried any other clients (except ssh/vim, of course).

Address tagging

You’ve probably seen email addresses that contain a plus sign in the local part. This is called address tagging. For example, the user has address userx@example.com; but to subscribe to different websites, she uses userx+hyves@example.com and userx+myspace@example.com. Mail to those tagged addresses are delivered to the original address as if the +hyves and +myspace parts don’t exist; no changes need to be made to the MDA configuration to create these addresses, which means the user may use as many different addresses (or more accurately ‘aliases’) as she wants, without needing the help of a sysadmin. The user can then use these tags to easily filter incoming mails, or to see which websites sell email addresses to spammers.

To enable address tagging, we only need to set 2 parameters in the Postfix configuration.
However, many websites use badly written regular expressions to verify email addresses, and do not allow the plus sign (violating RFC2822). To work around these poorly developed sites, I suggest we use the minus sign (‘-‘) instead; a dot (‘.’) could be another valid choice.

More recipient checking

I have some email addresses that I created a long time ago, but that I no longer use, and they just collect spam these days. So I’d like to reject all mail that is sent to those addresses. To this end, I created a file containing all addresses I’d like to blindly reject.

If I ever want to add addresses to this list, I add them to /etc/postfix/recipients, and I re-run the postmap command; I do not need to reload the Postfix config each time.

I could have done all this using the virtual database (see above). But the advantage of using a separate database is, that I can now run this check early in the process. No need to greylist these messages, and run them through a spam filter, if I know I’ll reject them anyway. Saves some resources.

Greylisting

Greylisting eliminates a considerable amount of spam. And it does so at low cost, because greylisting is light on resources. So this is a valuable technique to fight spam, especially if we use it before more costly techniques like a spam filter or Real-time Blackhole Lists.

You can have a look at Postgrey’s configuration in /etc/conf.d/postgrey, but there’s nothing that needs to be changed.
Postgrey runs as a daemon, so it must be added to the default runlevel:

root# rc-update add postgrey default
root# rc-service postgrey start

And then it can be added to the Postfix configuration, so that Postfix will query the daemon (bold and blue line was added):

Real-time Blackhole Lists

RBLs are lists of IP addresses known to belong to spammers. SMTP servers can check sender IP addresses against these lists by performing a DNS query.
You can add as many RBLs as you like, but be advised that each query takes time and resources.

Sender Policy Framework

SPF is not so much a technique to protect ourselves from receiving unwanted mail, as it is a technique to help others protect themselves from receiving unwanted mail sent in our name. And, obviously, we can protect ourselves from receiving unwanted mails by taking advantage of the SPF setup of others.

As I mentioned before, anyone can put any address as the sender address on any email.
SPF lets us create a DNS TXT record that defines which server(s) may be used to send mail for our domain. An example to make this more clear: if yellow.example.com is the only server we use to send mail onto the internet (the SMTP server we have configured in our email client), we create a TXT record saying so, allowing the receiver of an email which has our address as the sender address, to verify whether that email was sent from an SMTP server that we allow; if it wasn’t (our DNS record does not list the SMTP server that was used), the sender address is probably forged, and the mail is likely to contain spam or a virus.

The Postfix SPF policy daemon we’ve installed, tries to verify these DNS records for incoming mails. You will find a small configuration file at /etc/python-policyd-spf/policyd-spf.conf, but I didn’t need to change anything there.

For outgoing mails, all we need to do is add a correctly crafted TXT record to our own DNS. A sample record might look like this:

No extra steps need to be taken to reconfigure our SMTP server or our email client: the SMTP server on the receiving end of our emails will, if correctly configured, ask our DNS server to send the SPF TXT record, and will process it if it exists.

Remember to increase the Serial for the DNS zones, and to reload the Bind config.

root# rc-service named reload

You can verify the record with this command:

root# dig @127.0.0.1 example.com TXT

DomainKeys Identified Mail

DKIM is comparable to SPF in the sense that it allows a recipient of an email to verify the sender domain by means of a DNS record. Furthermore it signs an email, allowing the recipient to verify that the mail has not been modified between dispatch and retrieval. The signature is added to the headers of the email, while the public key to verify the signature can be found in a DNS TXT record.

This creates the files dkim.private and dkim.txt in all 3 subdirectories, where dkim.private contains the private key, and dkim.txt contains the DNS TXT record that holds the public key.
Problem is that I don’t know for which DNS server this record is generated, but Bind on my server wasn’t too happy with it.

The result may then be copied into the zone file for this domain. Do so for each key/domain. And don’t forget to increment the Serial in the zone file, and to reload the named config afterwards. Here is an example with both the SPF and DKIM records:

Send yourself an email, and look at its headers when it arrives.
Plug-ins/add-ons exist for email clients to verify DKIM signatures.

Spam filter

What I’ve described above, eliminates 90-95% of all the spam that is presented to my server. Obviously, spam is a very personal subject, as it depends at least in part of the mailing lists you’re subscribed to, the websites where you have accounts, and the amount of Viagra you order online. So your mileage may vary (I’ve always wanted to use that phrase 😉 ). But I think we can agree that we’ve done quite some spam filtering already, considering that we did it without a spam filter.

Now let’s send what’s left through Bogofilter, to filter out the final 5-10%.

Generally there are 2 points in the chain where you can send the mail through the spamfilter.
The first is the MTA (Postfix in our case). The advantage of this approach is that the MTA can reject spam at the door, letting the sender know that it’s useless to deliver spam to our server. The disadvantage is that all users share the same spam database, which means results are less personalized; additionally, this could be more expensive, because mails are tested against a database that contains tokens that are not relevant.
The other option is to let the LDA (Dovecot in our case) take care of the spam filtering. Disadvantage of this approach is that we cannot reject messages, but only discard. Advantages are that each user has their own, custom fit spam database, and that it’s easier to install and configure.

I decided to go for the latter approach.

We’ll do the filtering in 2 phases: first we’ll have Bogofilter add a header to the mail, indicating the probability of the mail being spam, and then we’ll use Sieve filters to decide where to send it.

Bogofilter needs a wordlist to compare the mails to. To create an empty wordlist, we’ll have to do this for each user (once):

The phrase ~userx (‘tilde-userx’) is a shortcut for writing /home/userx, but if you’ve changed the home directory to some other location, it still points to the correct directory.

Instead of sending the mail to Bogofilter directly, we’ll create a wrapper; this allows us to save the mail to a temporary file, which avoids forcing Bogofilter to keep the entire mail in memory while it is being processed.

First create a directory:

root# mkdir -p /usr/lib/dovecot/sieve/filter

And then save this as /usr/lib/dovecot/sieve/filter/sieve-bogofilter.sh:

You will need to compile the Sieve script into a binary. Normally the Sieve interpreter will do that for you, but it has no write permissions for this directory (and since it’s quite rare to create new scripts here, we won’t give it any).
We do this after editing the Dovecot configuration, because the vnd.dovecot.filter extension needs to be loaded.

root# sievec /usr/lib/dovecot/sieve/before/bogofilter.sieve

An alternative would be to call the script from the user’s Sieve scripts. On the one hand that would enable the more knowledgeable user to switch to some other spam filter (provided that an appropriate script is available in the sieve_filter_bin_dir), or to disable spam filtering all together. On the other hand, it would enable the less knowledgeable user to screw up the spam filtering when experimenting with their Sieve scripts, needing the help of a sysadmin to repair things.
For me, Bogofilter is the only installed spam filter (and currently I’m the only user), so this is a perfect solution. Even though I consider myself a rather knowledgeable user. 🙂
Anyway, to move the call to the user, all you need to do, is move the line ‘filter “sieve_bogofilter.sh”;‘ to the users’ Sieve scripts.

Reload the Dovecot configuration to activate the changes.

And finally we use the X-Bogosity header that was added, to file the mail into the correct folder. Which might look something like this:

The spamtest plugin translates the score given by Bogofilter (or any other spam filter) to a number between 0 — definitely not spam, or header line not found — to 10 — definitely spam — inclusive. And yes, this means that you could switch spam filters without users noticing; just change the sieve_before and the sieve_spamtest_status_header above.

Please do not be tempted to use the Sieve reject command for spam. Using reject will send a mail to the original sender of the mail, but since spammers often use forged sender addresses, your reject message will probably end up in the mailbox of somebody who had nothing to do with the spam you received. Read more about backscatter on Wikipedia.
Having spam rejected by Postfix is something entirely different, because in that case it is part of SMTP, where Postfix says to the sending server ‘No, I will not accept this mail.‘, resulting in the mail never entering the server. By the time the mail reaches the LDA, it has been accepted already, and by then the only way to ‘reject’ it, is to actually send a reply to the sender.

TL;DR:
The Sieve reject command should actually be called reply, and is not fit for rejecting spam.

Automated ham/spam (re-)classification

Well, we have a spam filter now. And it’s absolutely useless…

To determine whether a mail is spam, Bogofilter needs a list of words to compare the mail to. And currently, that list is still empty. Wouldn’t it be great if we had a mail folder where we could just drop mails to have them automatically classified as spam? Well, let’s create such mail folder.

Mind you: the directory name for a mail folder starts with a dot! (Which is kind of why your mail client doesn’t allow you to use dots in mail folder names.)
If you are the only user, you could also just create a new toplevel folder named ‘Spam‘ in your mail client.
You may need to restart your mail client to make the new folder visible. If it still doesn’t show after a restart, your mail client is probably configured to only display subscribed folders; you can find more information about that in your mail client’s documentation.

So, these are now (or will be) the folders in our mail account:

Spam
The folder we just created for mails of which we decide they are spam. The LDA will never deliver mail here; mails will ony enter manually. We will use this folder to train Bogofilter.

ProbablySpam
This folder will be created by our Sieve script for mails of which Bogofilter says they are probably spam. If they are, we move them to the Spam folder. If they are not, we move them to any other folder.

MaybeSpam
This folder will be created by our Sieve script for mails of which Bogofilter says they may be spam. If they are, we move them to the Spam folder. If they are not, we move them to any other folder.

other folders
For mails that are not spam. Mails that are not delivered to ProbablySpam or MaybeSpam will be delivered to one of the other folders, depending on our Sieve filters. If spam happens to fall through, we move it to the Spam folder.

Using ImapSieve we can attach hooks to all those folders for our ‘move’ actions.
These are the actions we need:

anywhere → Spam
Train as spam.

Spam → anywhere
Un-train as spam, train as ham.

MaybeSpam or ProbablySpam → anywhere except Spam
Train as ham.

anywhere → Trash
No training.

(In a next version of this document I will add some more actions. But for now this will have to do, because I feel I’m reaching the end of this page, and I’m really eager to publish. Writing this took a lot more time than it took to implement it.)

First, we create another directory:

root# mkdir /usr/lib/dovecot/sieve/pipe

And then we save this as /usr/lib/dovecot/sieve/pipe/bogofilter-train.sh:

This script expects an email on standard input, and 1 command line argument, which may start with a case-insensitive ‘s’ or ‘y’ to indicate spam, a case-insensitive ‘h’ or ‘n’ to indicate non-spam, or a case-insensitive ‘r’ or ‘u’ to indicate that the message should be retrained, because it was moved from the Spam box to elsewhere. It will then redirect it’s STDIN to Bogofilter.
Obviously, it needs to be executable:

root# chmod 755 /usr/lib/dovecot/sieve/pipe/bogofilter-train.sh

Then the Sieve scripts that will call that shell script.
Another directory, why not.

Like the Sieve script that checks the mails, the scripts here should also be compiled. And again, we cannot do this before we’ve made the changes to the Dovecot config, because the extensions and plugins won’t be available, yet.

And now you can start training your spam filter by moving mails around.
When you start with this setup, you will see that all mails are delivered to ProbablySpam. The more you train Bogofilter, the less mails will end up in ProbablySpam and MaybeSpam.
The first preparation step, at the top of this page, was to store your incoming spam in a separate folder. You can now move all of that to the Spam folder.
If you are 100% sure that your INBOX (or any other of your folders) is free of spam, you can do some ham training like this:

To train mails as spam instead of ham, you would use ‘-s‘ instead of ‘-n‘.

Actions that are currently missing or not yet investigated:

If I move a mail MaybeSpam|ProbablySpam → * → Spam, it is first learned as ham and then as spam, but it is never unlearned as ham, so my database is of by 1.

What happens when mail is saved by the mail client to ‘special’ mailboxes? Like Sent, Drafts, Archives, etc. I don’t think anything funny happens here, but I haven’t looked into it.

Some mail clients allow to save contacts/calendars on the IMAP server. What happens there?

Not an action, but still something to look into: I think all Dovecot configuration regarding Bogofilter could be moved to 1 single file, and there is no need to split it up into 90-sieve.conf, 90-sieve-extprograms.conf and 90-sieve-bogofilter.conf. I think all those plugin{} blocks end up merged, but I haven’t yet been able to find anything about this in the Dovecot documentation.

Virus scanner

ClamAV consists of 3 parts: clamd, which is the virus scanner, freshclam, which is the application that periodically updates the definitions, and clamav-milter, the interface between Postfix and ClamAV. All three are started at the same time with the clamd init-script. That is, if we enable all three:

Why the start and then the restart, you ask? Well, that’s because for me /usr/sbin/clamd refused to start the first time I tried. The log file said something about a file that could not be opened, but it didn’t say which file. So I figured it might be a file that freshclam should create, but hadn’t yet because it had just been started for the first time. And that’s why I restarted, and apparently I was right, because the second time it worked.
BTW: clamd takes quite some time to start, but in the end it will get there.

And to hook it into Postfix, we only need to add the milter to the Postfix config:

And we’re done. Send yourself a mail to verify that the headers are correctly added. Send yourself a virus to verify that it never arrives.

Webmail

What’s missing now for this otherwise rather complete, professional and future-proof mail system, is a webmail client. But since I really don’t need webmail, I’m going to leave that up to you, the reader; I know that several good webmail clients, that support Sieve message filtering, exist.

If you decide to install a webmail client, and to document the choice and installation, please let me know, and I’ll gladly link to your article. (The interesting part would probably be changing the SMTP/IMAP password from the webmail client.)

Message to users

Okay, this has nothing to do with any of the above. I just thought it was cool.
And you deserve a reward for scrolling, and hopefully reading, all the way down here.All work and no play makes Jack a dull boy.

Let’s create yet another directory:

root# mkdir /usr/lib/dovecot/post-login

Save this script as /usr/lib/dovecot/post-login/motd.sh, and make sure it’s executable:

Then, add this user’s mail aliases to /etc/postfix/virtual (remember to run `postmap‘ afterwards).

Add the username to the list of mail passwords:

root# echo "usery:" >> /etc/passwd.mail

(That’s 2 angle brackets! You will have a huge problem if you use only 1!)

And finally, the user should log in to the server using SSH, to set their mail password.

usery$ sudo mailpass

And this last bit really feels like a slack end, so again: if you decide to develop a small web interface for this, please let me know, so I can link to it.

And that’s it

Congratulations: your mail server is ready to be taken into production.

Changelog

2018-11-19
Initial publication.

About the author

Rob was a freelance Linux/*BSD sysadmin, software and web developer, and IT consultant for 15 years.
5 Years ago he retired to walk from his home in the Netherlands to Santiago de Compostela in Spain, and he has been drifting since, mainly in France.
Even though he swore he would never return to an office job, he would be open to small but interesting challenges that could be attacked from within a caravan somewhere in the south of France.