What this Document is About

This document covers how I have set up my Linux box to work as a Mail Server. The details will be specific to Arch Linux, but it will be be general enough for any *nix distribution if you have a knowledge of installing software packages, finding the location of configuration files, and starting/stoping services..

My goals in creating this mail system are to have all of my mail, from various email accounts, stored locally on my machine so I can access it anytime via SSH, the web, or imap. I wanted all of my mail delivered to my local user account and I wanted the spam to be sucked out and launched into space.

The backbone of the mail system is the Mail Transport Agent (MTA). It will handle receiving mail addressed directly to your machine, sending it to the correct users mail box, and sending mail out from the machine. I determined Postfix to be the best choice for my MTA and after some reading found Procmail for sorting mail, SpamBayes for detecting spam, and Fetchmail for grabbing mail from my pop mailboxes.

Bellow is a flow diagram for mail moving through the system:

The SpamAssassin and sa-learn can be ignored. I will describe how to use it with SpamBayes since I was not happy with Spam Assassin's performance and had made the diagram when I was still using it.

Software Used

Secure, fast, easy to administer drop in replacement for Sendmail. (MTA) Provides smtp-server.
(Requires sasl installed and support for sasl compiled into Postfix if sending mail through an intermediary as you may need to do if you have a domain name pointing to a dynamic IP address.)

A simple daemon written in Perl, to allow email relay control based on successful POP or IMAP logins.

Installing the Software

Installing Postfix

A note for those with dynamic IP addresses: Before installing Postfix we must first consider if it will need to make use of Simple Authentication and Security Layer (SASL). If you have a dynamic IP address and are using a service like dyndns.org you will need to have Postfix send your mail through your ISP's mail server (with authentication) making use of SASL for the connection. This is because many domains that you will send email to will recognize your hostname as pointing to a dynamic IP address and send the mail back. Check if your distributions Postfix package was built with SASL support or if you are compiling Postfix from scratch add SASL with a ./configure option.

Installing Mutt

$ pacman -S mutt

Installing courier-imap and courier-authlib

SquirrelMail as well as other mail clients (evolution, outlook, cell phone, etc.) will use this to connect to the mail box. Courier-imap also depends on courier-authlib, so this command will install that as well.

$ pacman -S courier-imap

Installing SquirrelMail

$ pacman -S squirrelmail

Installing pop-before-smtp

You are going to want to consider using Postfix/TLS or pop-before-smtp if you are interested in users connected via IMAP being able to send mail (using your mail server as a SMTP relay). I recommend using Postfix/TLS but will not cover how to install/configure it in this document.

Pop-before-smtp is not in pacman repositories. Substitute version numbers with the most current.

Read the README in this directory and then read README.QUICKSTART in the contrib directory for where to copy the necessary files to.

Postfix Configuration

main.cf

Now change to the '/etc/postfix' directory and open up main.cf for editing.

The domain name for my machine is through dyndns.org. Set your myhostname and my domain to something like this:

myhostname = example.com
mydomain = example.com

Next we are going to want to set Postfix up for using qmail-style delivery with Maildir (where each email is stored in its own file). This is for courier-imap which SquirrelMail will be using to access the mail and will have other advantages. Don't forget the ending forward slash.

home_mailbox = Maildir/

Set it up so that incoming mail is sent through procmail and so we can alias names/users to mailboxes.

mailbox_command = /usr/bin/procmail -a "$DOMAIN"

alias_maps = hash:/etc/postfix/aliases

If you have a dynamic IP or have trouble with mail being returned from large ISP's or mail systems you are going to have to take a few extra steps. We will set it up so that mail will go through a relay host. In this example I will use my ISP's (SBC) smtp server as an example. Make these additional changes to main.cf

That wraps up our changes to main.cf. We just need to create the hash aliases databases that we defined above. Add these lines to the top of /etc/postfix/aliases (replacing hypexr with your user account of course):

root: hypexr
spam: hypexr
ham: hypexr

Generate the database. From /etc/postfix dir:

$ postalias aliases

Set up smtp server and authentication for SBC mail relay. sasl_passwd:

smtp.sbc.mail.yahoo4.akadns.net my_username@sbcglobal.net:my_passw

Note:Server name can change from time to time. There is a CNAME that points to this name but CNAMEs do not work here.

Generate the database:

$ ./postmap sasl_passwd

Start the postfix server and we are done with postfix!!!!

$ /etc/rc.d/postfix start

Having problems getting it to start? Check out the Postfix log file usually in /var/log/mail or /var/log/maillog.

Fetchmail Configuration

We are going to run our fetchmail daemon from user root. So create the file /root/.fetchmailrc:

set postmaster "postmaster"
set bouncemail
set no spambounce
set properties ""
set daemon 60
poll pop.example.com with proto POP3
user 'user_name' there with password 'my_password' is 'local_user_to_deliver_to' here options flush
poll pop.gmail.com with proto POP3 port 995
user 'my_gmail_un@gmail.com' there with ssl with password 'my_password' is 'hypexr' here options keep

This tells fetchmail to check every minute for mail at pop.mypop.com and also at my gmail account. The flush option at the end of the mypop entry tells it to remove the mail after it is downloaded. I do not want the mail removed from my Gmail account so I use the keep option. Google also uses ssl, hence the "with ssl". Fetchmail comes with its configuration utility fetchmailconf which makes setting up your .fetchmailrc file easy.

I don't believe that Arch Linux has a rc script for fetchmail. The one that I made (/etc/rc.d/fetchmail) looks like this (stat_busy, stat_fail, stat_done are specific to Arch Linux. You could take these out):

Or if you don't want to create/modify an rc script you can start the fetchmail daemon like:

$ /usr/bin/fetchmail -d 60

Spambayes and Procmail Configuration

We are going to be using Spambayes to train and determine if mail is spam and Procmail to sort the mail into folders when they enter our mail system. These two go together as far as Procmail is going to use Spambayes to help sort the mail.

I had used SpamAssassin for a year and had not been very happy with its performance. I have only been using Spambayes for a few weeks and have been pleasantly surprised by how well it has worked and how quickly it adapts. In fairness I had been training SpamAssassin on all of my spam, the spam it had caught and the spam it had missed as well. It was also an older version and I was not using all of the features. With that being said if you are curious give it a try. Many other excellent programs exist. A quick search on Google will turn them up.

When Spambayes is installed it is going to move several files that start with sb_ into your /usr/bin directory. We are going to be interested in sb_mboxtrain.py and sb_filter.py. The former trains the spam filter and the later classifies incoming spam. There are two methods I tried for the initial training:

Since I already had thousand of spam and ham (non spam) messages saved up I first tried initially training on this older mail. The results were okay but it was miss sorting mail. These problems would have sorted them selves out with the continued training of the misclassified mail but it would have also took a little while because the database had already been trained so much.

The second method I used was extremely effective. I started out with no training on initial messages. I simply let Spambayes work with no information. I then sent the spam and ham messages that ended up in the wrong mailbox to a designated folder. I then run the Spambayes training program on these miss classified messages.

The results from the second method have been amazing. Without even having to manually classify very many messages Spambayes has been performing at superior levels. Check out Spambayes Wiki for many other training methods. I will discuss easy methods of sorting miss classifications in the Mutt and Squirrel Mail sections.

Spam System Initial Setup

If the standard Maildir directory (~/Maildir) doesn't already exist lets create it with the necessary sub directories to make it a mail box. Run these commands from your user account who will be receiving the mail.

$ mkdir ~/Maildir
$ mkdir ~/Maildir/{new,cur,tmp}

And now the other 4 directories that we are going to use for dealing with spam (Making the directories hidden is going to be done for integration with Squirrelmail):

The messages that we are going to be training as spam are going to be in ~/Maildir/.MissedSpam and ham will be in ~/Maildir/.MissedHam. Lets create the script and cron job that will take care of training the Spambayes db. Create a file ~/Maildir/train_spambayes and add this:

#!/bin/sh
# Script to copy mail missed spam and ham into correct folders
# and run sb_mboxtrain.py to train spambayes
# Training will be done only on missed spam and ham
# Files we saved or bounced may be in the new directory
# We want them in cur for training
mv /home/hypexr/Maildir/.MissedSpam/new/* \
/home/hypexr/Maildir/.MissedSpam/cur 2>/dev/null
mv /home/hypexr/Maildir/.MissedHam/new/* \
/home/hypexr/Maildir/.MissedHam/cur 2>/dev/null
/usr/bin/sb_mboxtrain.py -d /home/hypexr/.hammie.db \
-g /home/hypexr/Maildir/.MissedHam \
-s /home/hypexr/Maildir/.MissedSpam

Make the script executable:

$ chmod +x ~/Maildir/train_spambayes

Set up our database that will be consulted for new email and trained on new spam/ham:

$ /usr/bin/sb_filter.py -d $HOME/.hammie.db -n

You will want cron installed so that it can launch the script above, which trains Spambayes, every night at 3:21 am. Edit your cron jobs with the command crontab -e and add:

21 3 * * * ~/Maildir/train_spambayes

Procmail Configuration for Spambayes

These are basic procmail recipes that will sort the spam from ham. From Spambayes on Unix or Linux. Add these lines to your mail recipient's ~/.procmailrc:

So incoming mail is not filtered through procmail's rules where Spambayes's sb_filter decides whether it is spam or ham. The mail is now sitting in the correct directories and ready to be accessed by whatever reader/front-end you would like. Next I going to describe setting up Mutt and SquirrelMail for viewing your mail. Mutt will be the most powerful option combining speed and sleekness but SquirrelMail will be able to match functionality.

Mutt Configuration

First we need to set the MAIL variable so that Mutt will know where to look for your mail. In your ~/.bashrc (create if does not exist) add the line:

MAIL='~/Maildir'

Exit out of your shell and log back in so that the variable will be set or run export MAIL='~/Maildir'. On some systems mutt will not use $MAIL to determine the location of your mail box. When trying to start mutt if it reports a message that no mail directory exists you can try setting the MAILDIR variable or just create the directory that it wants to use and make it a valid Maildir: mkdir -p ~/.maildir/{cur,new,tmp} where .maildir is the folder that mutt wants to use. If you use a directory other than ~/Maildir remember to substitute that wherever I reference Maildir in this document.

Now for the fun part. Lets make everything look pretty and integrated in Mutt. If you start up Mutt right now you will see any mail that your system has received since you have completed the steps above. If no mail shows up in Mutt and there should be mail you need to first check if Spambayes has filtered it into your .Unsure or .CaughtSpam directories. Type 'c' to change directories and enter ~/Maildir/Unsure and check out the mail thats been classified as spam: 'c' ~/Maildir/CaughtSpam. If you still have not found the mail that you are expecting to have received make sure that you have configured Postfix, Fetchmail, and Procmail correctly. Also, try sending mail directly to your_user_accunt@example.com and see if it shows up in one of the directories above.

Now mail is flowing into your system and we can check how Spambayes has classified it by checking the different mail folders. The miss classified mail just needs to be moved into the correct folders for the training. To manually move the misclassified spam mail:

From you Maildir inbox tag each message that has arrived that is spam. Do this by pressing 't' and an asterisk will appear by the messages. After all of the messages have been tagged, press '; m' and enter ~/Maildir/MissedSpam and the spam messages in your inbox will be moved to the MissedSpam mailbox for training by spambayes.

When looking at the mail that it has classified as spam in '~/Maildir/CaughtSpam', tag all of the messages it considered spam that were ham and press '; C' and enter ~/Maildir/MissedHam. Now tag them again and press '; m' and enter ~/Maildir to move them into your inbox, where they should have been placed in the first place.

Of course all of this manual stuff is a huge pain so lets automate all of it in .muttrc.

.muttrc Configuration

This .muttrc file is going to save you a quazillion.. thats right a quiazalliona hours in productivity in dealing with our current setup. Here it is:

# Emacs is my editor of choice. I have mostly read that people like
# using emacs-client here and using an emacs server so that the
# editing opens up in an existing emacs session but I prefer
# loading up a fresh instance. The default editor is some 'vi'
set editor="emacs"
# Setting these macros is going to save us all of that time I was
# talking about. Now instead of having to type all of those
# characters to move the mail around we will can just tag the messages
# and, for the instance of spam in your inbox, hit 'S'.
# Move mail to correct directories
macro index S ";s~/Maildir/.MissedSpam\r\r$\r"
macro pager S ";s~/Maildir/.MissedSpam\r\r$\r"
macro index H ";C~/Maildir/.MissedHam\r\r;s~/Maildir\r\r$\r"
macro pager H ";C~/Maildir/.MissedHam\r\r;s~/Maildir\r\r$\r"
macro index M ";s~/Maildir/\r"
macro pager M ";s~/Maildir/\r"
# This one lets me quickly move job search related emails to the
# correct directory. Want to give me a job? :)
macro index B ";s~/Maildir/.Job\r\r$\r"
macro pager B ";s~/Maildir/.Job\r\r$\r
# This is how we are going to move around to the different
# mailboxes. Hitting 'alt-1' will take us to our inbox
# 'alt-2' and we are looking at the mail Spambayes classified
# as spam. etc.
# The extra mailboxes I have in here are for mailing lists I am on.
# I will show the procmail recipe that automatically puts
# the mailing list mail into its correct mailbox further down.
# Move to mailboxes quickly
macro index <Esc>1 "c~/Maildir\r"
macro pager <Esc>1 "c~/Maildir\r"
macro index <Esc>2 "c~/Maildir/.CaughtSpam\r"
macro pager <Esc>2 "c~/Maildir/.CaughtSpam\r"
macro index <Esc>3 "c~/Maildir/.Unsure\r"
macro pager <Esc>3 "c~/Maildir/.Unsure\r"
macro index <Esc>4 "c~/Maildir/.Tur\r"
macro pager <Esc>4 "c~/Maildir/.Tur\r"
macro index <Esc>5 "c~/Maildir/.Sbayes\r"
macro pager <Esc>5 "c~/Maildir/.Sbayes\r"
macro index <Esc>6 "c~/Maildir/.Wikka\r"
macro pager <Esc>6 "c~/Maildir/.Wikka\r"
# Training Maildirs for Spam and Ham
macro index <Esc>8 "c~/Maildir/.MissedSpam\r"
macro pager <Esc>8 "c~/Maildir/.MissedSpam\r"
macro index <Esc>9 "c~/Maildir/.MissedHam\r"
macro pager <Esc>9 "c~/Maildir/.MissedHam\r"
# Job E-mails
macro index <Esc>0 "c~/Maildir/.Job\r"
macro pager <Esc>0 "c~/Maildir/.Job\r"
# Setting these colors makes it easy to tell which emails
# Spambayes has missed.
# Turn spam red and unsure green
color index red default "~h '^X-Spambayes-Classification: spam'"
color index green default "~h '^X-Spambayes-Classification: unsure'"
# Turn gmail mail brightblue
# (My regular expression is not correct here. I think that it
# is looking anywhere in the header for my gmail address
# instead of what I wanted.. Just in the 'To:'.
# It still works for the most part)
color index brightblue default "~h '(^)*my_name@gmail.com'
# Lets set more colors to make things look beautiful. Judging by
# my title for this section I must have had bad feelings towards
# colors when I added this.
#Color crap
color index brightwhite default ~N # color for new messages
color status black yellow
color attachment brightyellow default # file attachments
color search brightred default # search matches
color quoted brightyellow default # quoted text in replies
color quoted1 magenta default # quoted text in replies
color body cyan default "((ftp|http|https)://|news:)[^ >)\"\t]+" # URLs
color body cyan default "[-a-z_0-9.+]+@[-a-z_0-9.]+" # email
# The rest of this stuff is pretty important for functionality and
# readability.
# Don't ask to move read message
set move=no
# My signature is in this file
set signature="~/.signature"
# Change mail to look like from scott@hypexr.org
# Stick your email address here.
my_hdr From: Scott Rippee <scott@hypexr.org>
# Header control h displays header when in pager
# I want to only see the unignored by default
ignore * # weed out all headers
unignored date from: to cc subject organization # now show me these...
# Thanks to many people whom have written the documents and posts that I
# have gotten these settings from.

New Mutt Commands

Here are the most useful (as far as dealing with spam) commands that we can now use in mutt:

So for example, if a couple of spam messages show up in your inbox you can now tag them and hit 'S' (remember its uppercase s) and have them vanished into .MissedSpam for training.

SquirrelMail Configuration

To configure SquirrelMail go into your html_base_directory/squirrelmail/config and run ./conf.pl. Edit the options to fit your needs. If you want to test your config cruise over to http://localhost/squirrelmail/src/configtest.php.

Now go to http://localhost/squirrelmail/ and login using your user account. If you have any email in your in box you should see it after you log in. Now click on the Folders option that is along the top, select all of the mailbox folders that we created earlier (they are listed here because we prefixed them with a period), and click on subscribe. Hit refresh and the mailboxes should be listed in the side panel. Now you can use SquirrelMail for almost all of you mail needs including tagging messages and moving them to the correct folders for Spambayes training and checking on the mail that Spambayes has classified as spam and unsure. The only functionality from our mutt setup that can't be duplicated in SquirrelMail is copying missed ham into its training folder and then moving it into your inbox. This is due to the lack of copying ability. So you will have to go into the ham training folder to see some of the mail that should be in your inbox.

!!! IMPORTANT SECURITY NOTE!!! - Using SquirrelMail like this is not secure! Sending the plain text passwords that you use for your Unix account can be accessed by by a 3rd party, which would them access to your account. You should set up a secure server with SSL and have Courier-imap use passwords that differ from the users system password using Courier Authlib, which I will demonstrate in the next session. Excellent documents exist all over the net for setting up your web server with ssl.

Courier Authentication

Now is a good time to address a bit of essential security. As of now when a user connects via the SquirrelMail or IMAP interface the passwords are being sent in plain text and if these are sniffed that users system account is compromised. Without adding secure connections we can minimize the threat by using alternate passwords for imap connections. The snoop would then have access to your mail, but not be able to log in to the system.

Edit your authdaemonrc file. If your on Arch Linux this will be in /etc/authlib. Under authmodulelist I commented out:

authmodulelist="authpam authpwd authuserdb authshadow ..."

And replaced it with:

authmodulelist="authuserdb"

I believe that adding authuserdb to the front of the list may work, but I am only using this method of authentication so it is all I need in my list.

You will now be prompted for this pseudo users new password. Next make sure that /etc/authlib/userdb has read/write/execute access for the owner. If not chmod 700 /usr/authlib/userdb. Now create the user database that courier is going to look to for authentication:

$ makeuserdb

Again look to using Postfix/TLS for a tunneling the connection through a secure connection.

Pop-before-smtp Configuration

To successfully configure pop-before-smtp follow the straight forward instructions in contrib/README.QUICKINSTALL. When you are done and everything is working it is important to make sure that you have not some how made your mail server an open relay! This would cause your mail server to be flooded with traffic and be put on the open relay ban lists.

!!! IMPORTANT SECURITY NOTE!!! - I will not use the pop-before-smtp method because it involves sending a password over an insecure connection. Use Postfix through a SSL connection, Postfix/TLS, to accomplish this. Since your local machines are likely already allowed to use your Postfix as a mail relay the only time that you would be using this method is from a non-trusted server where you password can be sniffed. If you still choose to use this method make sure that you have changed the users courier password as described in Courier Authentication.

Additional Simple Procmail Recipes and Mailbox Config

If you are a member of any news letters/mailing lists you can have them automatically sorted to their own mailbox using Procmail. First create the new mail directory in your ~/Maildir directory. Lets call it .Sbayes and pretend that we are on the Spambayes mailing list. So exactly as before with the spam related mailboxes:

In the .mutrc example above it shows an example of setting alt-5 as the macro to see your Spambayes emails.

A slightly different method can be used to send spam and ham for training. Adding the following lines will allow you to bounce spam messages to spam@localhost (localhost or your mail_domain.com) and ham messages to ham@localhost:

If you want to bounce messages to the spam and ham addresses like this, you will need to make sure that the headers stay intact. Do not try using this or a similar technique for forwarding mails to an address because the email's headers would not be intact and this is a large part of the spam classification. This method works because above we specified spam and ham as aliases for hypexr in the Postfix aliases file.

Conclusion

For Arch Linux users, Add saslauthd authdaemond courier-imap postfix fetchmail to DAEMONS in your /etc/rc.conf file and they will be started automatically when your machine boots. If you are running a web server have the httpd service listed after the services we are adding.

Enjoy your new mail system and centralized mailbox. If you are handling a large number of users for you email system this is just the tip of the iceberg and you are going to mostly be interested in virtual users, security, and a system wide spam classifier. There are many good documents on various setups to accomplish this, see the links section.