Relaying with TLS in Sendmail

This document describes how to setup the Sendmail Mail Transport Agent (MTA) to allow encrypted email via Transport Layer Security (TLS). Roaming client systems such as laptops running Unix can relay mail through the static servers using this setup. TLS relaying works well for remote relaying of systems, or where a x509 certificate infrastructure is in place. To allow remote relaying for users, SMTP AUTH is usually a better fit.

In the following example, a sendmail 8.12 client is configured to talk with a sendmail 8.12 server via TLS. The steps are to get STARTTLS support in sendmail, tell sendmail where to look for certificate files, and finally to generate and setup the required TLS certificates.

These notes assume one is reasonably competent with Unix, Sendmail, and OpenSSL. For debugging purposes, the values of various Sendmail macro may need to be logged. This can be done via a custom ruleset:

If not, recompile sendmail with STARTTLS support via a custom site.config.m4. OpenSSL must be installed on the system in question first. Alternatively, STARTTLS may be available in a special package or port of sendmail, depending on the vendor in question.

Once STARTTLS support is compiled into sendmail, various TLS parameters will needed to be added to the sendmail configuration file. These parameters tell sendmail where to find the certificates needed for TLS, among other things.

Add the following to the sendmail.mc file for sendmail.cf, and the equivalent submit.mc for the sendmail 8.12 client system(s). Then rebuild the *.cf files.

The sendmail client may need group readable permissions set of the key file, if both the sm-mta (listening to localhost, for example) and the sm-msp (for locally submitted mail) are employed. Set the following in the submit.mc.

define(`confDONT_BLAME_SENDMAIL', `GroupReadableKeyFile')

The key file will need to be readable by root (for sm-mta) and smmsp (for sm-msp).

Some sites block or redirect outgoing mail at port 25 for various reasons; to avoid this, have the server listen at port 587 (recent sendmail do this by default), and direct sm-msp mail to this port via the following definition of msp in the submit.mc file.

FEATURE(`msp', `mail.example.org', `MSA')

On my laptop, I do not run any sendmail daemons. I use a script to manually run the queue if things have been piling up (only when I send e-mail with the network down or my server is unavailable for some reason), which essentially runs the following command.

Generating the required certificates is the most difficult part: there are several different ways the certificates can be generated, the resulting certificates must be properly installed where sendmail can find them, and portions of the certificate data must be translated into sendmail?s access map file on the server.

For my main server, I purchase a certificate from a third-party Certificate Authority (CA) to allow other mail clients and servers to easily verify my system. For internal hosts like my laptop, I run my own CA and generate certificates as needed.

The signing certificate(s) for the system certificate(s) in question will need to be present in the confCACERT file, or be in proper hashed format in the confCACERT_PATH directory. Scripts can assist in the task of generating system certificates or creating the certificate hash format used by OpenSSL.

To support a new client, I generate a Certificate Signing Requests (CSR) from the certs directory of my client configuration, and sign the request in the CA area for my site, then save the resulting certificate as host.cert. I have updated the system-wide openssl.cnf to use defaults for where I am.

# cd /etc/mail/certs# make csrGenerating RSA private key, 1024 bit long modulus.........................++++++..................................++++++e is 65537 (0x10001)Using configuration from /System/Library/OpenSSL/openssl.cnfYou are about to be asked to enter information that will be incorporatedinto your certificate request.What you are about to enter is what is called a Distinguished Name or a DN.There are quite a few fields but you can leave some blankFor some fields there will be a default value,If you enter '.', the field will be left blank.-----Country Name (2 letter code) [US]:State or Province Name (full name) [Washington]:Locality Name (eg, city) [Seattle]:Organization Name (eg, company) [Sial.org]:Organizational Unit Name (eg, section) []:Common Name (eg, YOUR name) []:legacy.sial.orgEmail Address []:jmates@sial.org

To allow relaying by TLS clients, the access map will need to be populated on the server with entries allowing relaying for certificates matching the specified certificate data. This check will only be done for certificates that have a {verify} macro status of OK: that is, when sendmail is able to validate the certificate of the client in question.

Import signing certificates.

To verify, sendmail must know about the signing certificate. This means any custom CA certificates must be imported into either the confCACERT or properly hashed in the confCACERT_PATH directory. Using my setup, I copy the cacert.pem from my CA area and save it as exampleca.cert under the /etc/mail/certs/CA directory on my mail systems, then use the make links to create the proper hash link OpenSSL requires.

This will need to be done for all custom certificates, whether from a private CA, a self-signed certificate, or a third-party CA certificate not present in the global confCACERT file. Use of the confCACERT_PATH directory is encouraged, as too many entries in the global file may cause OpenSSL to not work as expected.

Server access map setup.

In the access map file, add CERTIssuer (or additionally CERTSubject) entries and rebuild the hashed version of access with makemap. The following involved example shows how to extract the x509 issuer data from the certificate /etc/mail/certs/CA/exampleca.cert with openssl and encode it for use in sendmail with perl.

The above line can then be appended to the access map and the map file rebuilt to allow relaying for all certificates signed with the matching certificate. Ideally, it should be encapsulated into a script or Makefile to hide the dirty work that needs to be done.

If things do not work the first time (or second, third, and subsequent tries), remember that log files are your friend. And that embarrassingly simple typos will often be spotted by someone else reviewing your work.

I have seen the openssl output change somewhere between the 0.9.6 and 0.9.7 releases (?) with the email attribute changing from Email to emailAddress, which caused the exact match access map entries to fail. Regenerating the access map entries from the certificate data solved the problem.

An alternative to the access map CERTIssuer and CERTSubject relaying is to create a custom ruleset that allows relaying for certain certificates. In the following example, the md5 fingerprints of the client certificates will be used to allow relaying. This method is better suited to environments that lack central certificate authority certificates or a key signing infrastructure.

These instructions assume various steps outlined above have already been taken care of, for example STARTTLS support in sendmail.

Create server relay ruleset.

A custom ruleset will need to be created on the server to allow relaying by certificate fingerprints. In the sendmail.mc file, add the following at the bottom. It allows relaying should the client certificate presented be verified (exist locally on the server) and the md5 fingerprint of the certificate exist with a RELAY value in the hashed md5map file.

Only the fingerprint of the client certificate will be available, not of any signing certificate. This information will need to be obtained from every client certificate that will be allowed to relay, which may not scale well with large numbers of clients.

Each client certificate (host.cert using my standard setup) will also ideally need to be transferred to the server; this step is necessary to allow the server to properly verify the certificate in question. Client certificates should be stored in the confCACERT_PATH directory in the proper hash-as-name format. Otherwise, remove the {verify} macro ruleset checks from the ruleset above. Not verifying the certificate would in theory allow an attacker to generate a different certificate that shares the same md5 of a certificate in use to relay through the system in question.

At minimum, a TLS_Srv entry for the outgoing e-mail server mail.example.org will need to be made. However, someone could create a malicious MX record that points to a different server. Thus, a TLS_Rcpt entry is needed to ensure a particular certificate is used by the server. The following rules show how to verify a named server as well as ensure the server uses a verified certificate that matches the named certificate issuer data.