Postfix After-Queue Content Filter

Introduction

This document requires Postfix version 2.1 or later.

Normally, Postfix receives mail, stores it in the mail queue
and then delivers it. With the external content filter described
here, mail is filtered AFTER it is queued. This approach decouples
mail receiving processes from mail filtering processes, and gives
you maximal control over how many filtering processes you are
willing to run in parallel.

The after-queue content filter is meant to be used as follows:

Network or local users

->

Postfix queue

->

Content filter

->

Postfix queue

->

Network or local mailbox

This document describes implementations that use a single
Postfix instance for everything: receiving, filtering and delivering
mail. Applications that use two separate Postfix instances will
be covered by a later version of this document.

The after-queue content filter is not to be confused with the
approach that is described in the SMTPD_PROXY_README document,
where incoming SMTP mail is filtered BEFORE it is stored into the
Postfix queue.

This document describes two approaches to content filter
all email, as well as several options filter mail selectively:

An external content filter receives unfiltered mail from Postfix
(as described further below) and does one of the following:

Re-inject the mail back into Postfix, perhaps after changing
content and/or destination.

Reject the mail (by sending a suitable status code back to
Postfix). Postfix will return the mail to the sender.

NOTE: in this time of mail worms and forged spam, it is a VERY
BAD IDEA to send viruses back to the sender address, because the
sender address is almost certainly not the originator. It is better
to discard known viruses, and to quarantine material that is
suspect so that a human can decide what to do with it.

The first example is simple to set up. Postfix receives
unfiltered mail from the network with the smtpd(8) server, and
delivers unfiltered mail to a content filter with the Postfix
pipe(8) delivery agent. The content filter injects filtered mail
back into Postfix with the Postfix sendmail(1) command, so that
Postfix can deliver it to the final destination.

Line 21: The idea is to first capture the message to
file and then run the content through a third-party content filter
program.

Line 22: If the mail cannot be captured to file, mail
delivery is deferred by terminating with exit status 75 (EX_TEMPFAIL).
Postfix places the message in the deferred mail queue and tries
again later.

Line 25: You will need to specify a real content filter
program here that receives the content on standard input.

Line 26: If the content filter program finds a problem,
the mail is bounced by terminating with exit status 69 (EX_UNAVAILABLE).
Postfix will return the message to the sender as undeliverable.

Note: in this time of mail worms and spam, it is a BAD
IDEA to send known viruses or spam back to the sender, because that
address is likely to be forged. It is safer to discard known to be
bad content and to quarantine suspicious content so that it can
be inspected by a human being.

Line 28: If the content is OK, it is given as input to
the Postfix sendmail command, and the exit status of the filter
command is whatever exit status the Postfix sendmail command
produces. Postfix will deliver the message as usual.

Line 30: Postfix returns the exit status of the Postfix
sendmail command.

I suggest that you first run this script by hand until you are
satisfied with the results. Run it with a real message (headers+body)
as input:

% /path/to/script -f sender recipient... <message-file

Once you're satisfied with the content filtering script:

Create a dedicated local user account called "filter". This
user handles all potentially dangerous mail content - that is
why it should be a separate account. Do not use "nobody", and
most certainly do not use "root" or "postfix".

Create a directory /var/spool/filter that is accessible only
to the "filter" user. This is where the content filtering script
is supposed to store its temporary files.

Configure Postfix to deliver mail to the content filter
with the pipe(8) delivery agent.

This runs up to 10 content filters in parallel. Instead of a
limit of 10 concurrent processes, use whatever process limit is
feasible for your machine. Content inspection software can gobble
up a lot of system resources, so you don't want to have too much
of it running at the same time.

The "content_filter" line causes Postfix to add one content
filter request record to each incoming mail message, with content
"filter:dummy". This record overrides the normal mail routing
and causes mail to be given to the content filter instead.

The content_filter configuration parameter accepts the same syntax
as the right-hand side in a Postfix transport table.

Execute "postfix reload" to complete the change.

To turn off content filtering, edit the master.cf file, remove
the "-o content_filter=filter:dummy" text from the entry that
defines the Postfix SMTP server, and execute another "postfix
reload".

With the shell script as shown above you will lose a factor of
four in Postfix performance for transit mail that arrives and leaves
via SMTP. You will lose another factor in transit performance for
each additional temporary file that is created and deleted in the
process of content filtering. The performance impact is less for
mail that is submitted or delivered locally, because such deliveries
are already slower than SMTP transit mail.

The problem with content filters like the one above is that
they are not very robust. The reason is that the software does not
talk a well-defined protocol with Postfix. If the filter shell
script aborts because the shell runs into some memory allocation
problem, the script will not produce a nice exit status as defined
in the file /usr/include/sysexits.h. Instead of going to the
deferred queue, mail will bounce. The same lack of robustness can
happen when the content filtering software itself runs into a
resource problem.

The simple content filter method is not suitable for content
filter actions that are invoked via header_checks or body_checks
patterns. These patterns will be applied again after mail is
re-injected with the Postfix sendmail command, resulting in a mail
filtering loop. The advanced content filtering method (see below)
makes it possible to turn off header_checks or body_checks patterns
for filtered mail.

The second example is more complex, but can give better
performance, and is less likely to bounce mail when the machine
runs into some resource problem. This content filter receives
unfiltered mail with SMTP on localhost port 10025, and sends filtered
mail back into Postfix with SMTP on localhost port 10026.

The example given here filters all mail, including mail that
arrives via SMTP and mail that is locally submitted via the Postfix
sendmail command. See examples near the end of this document for
how to exclude local users from filtering, or how to configure a
destination dependent content filter.

You can expect to lose about a factor of two in Postfix
performance for mail that arrives and leaves via SMTP, provided
that the content filter creates no temporary files. Each temporary
file created by the content filter adds another factor to the
performance loss.

Advanced content filter: requesting that all mail is filtered

To enable the advanced content filter method for all mail,
specify in main.cf:

The "content_filter" line causes Postfix to add one content
filter request record to each incoming mail message, with content
"scan:localhost:10025". The content filter request records are
added by the smtpd(8) and pickup(8) servers (and qmqpd(8) if you
decide to enable this service).

Content filter requests are stored in queue files; this
is how Postfix keeps track of what mail needs filtering. When a
queue file contains a content filter request, the queue manager
will deliver the mail to the specified content filter regardless
of its final destination.

The "receive_override_options" line disables address
manipulation before the content filter, so that the content filter
sees the original mail addresses instead of the result of virtual
alias expansion, canonical mapping, automatic bcc, address
masquerading, etc.

To turn off content filtering, delete or comment out the two
above main.cf lines. All other changes made for advanced content
filtering have no effect when content filtering is turned off.

This runs up to 10 content filters in parallel. Instead
of a limit of 10 concurrent processes, use whatever process limit
is feasible for your machine. Content inspection software can
gobble up a lot of system resources, so you don't want to have too
much of it running at the same time.

With "-o smtp_send_xforward_command=yes", the scan transport
will try to forward the original client name and IP address to the
after-filter smtpd process, so that filtered mail is logged with
the real client name IP address. See smtp(8) and XFORWARD_README
for more information.

Advanced content filter: running the content filter

The content filter can be set up with the Postfix spawn service,
which is the Postfix equivalent of inetd. For example, to instantiate
up to 10 content filtering processes on localhost port 10025:

"filter" is a dedicated local user account. The user will
never log in, and can be given a "*" password and non-existent
shell and home directory. This user handles all potentially
dangerous mail content - that is why it should be a separate account.

If you want to have your filter listening on port localhost:10025
instead of Postfix, then you must run your filter as a stand-alone
program, and must not use the Postfix spawn service.

Advanced filter: injecting mail back into Postfix

The job of the content filter is to either bounce mail with a
suitable diagnostic, or to feed the mail back into Postfix through
a dedicated listener on port localhost 10026.

The simplest content filter just copies SMTP commands and data
between its inputs and outputs. If it has a problem, all it has to
do is to reply to an input of `.' from Postfix with `550 content
rejected', and to disconnect without sending `.' on the connection
that injects mail back into Postfix.

These receive override options are either implemented by the
SMTP server itself, or they are passed on to the cleanup server.

The "-o smtpd_xxx_restrictions" and "-o mynetworks=127.0.0.0/8"
override main.cf settings. They turn off junk mail controls that
would only waste time here.

With "-o smtpd_authorized_xforward_hosts=127.0.0.0/8",
the scan transport will try to forward the original client name
and IP address to the after-filter smtpd process, so that filtered
mail is logged with the real client name and IP address. See
XFORWARD_README and smtpd(8).

With the "sandwich" approach to content filtering described
here, it is important to match the filter concurrency to the
available CPU, memory and I/O resources. Too few content filter
processes and mail accumulates in the active queue even with low
traffic volume; too much concurrency and Postfix ends up deferring
mail destined for the content filter because processes fail due to
insufficient resources.

Currently, content filter performance tuning is a process of
trial and error; analysis is handicapped because filtered and
unfiltered messages share the same queue. As mentioned in the
introduction of this document, content filtering with multiple
Postfix instances will be covered in a future version.

After this, you can follow the same procedure as outlined in
the "advanced" or "simple" content filtering examples above, except
that you must not specify "content_filter" or "receive_override_options"
in the main.cf file.

If you are an MX service provider and want to apply different
content filters for different domains, you can configure ONE Postfix
instance with multiple SMTP server IP addresses in master.cf. Each
address provides a different content filter service.

After this, you can follow the same procedure as outlined in
the "advanced" or "simple" content filtering examples above, except
that you must not specify "content_filter" or "receive_override_options"
in the main.cf file.

Set up MX records in the DNS that route each domain to the
proper SMTP server instance.

You can do this in smtpd access maps as well as the cleanup
server's header/body_checks. This feature must be used with great
care: you must disable all the UCE features in the after-filter
smtpd and cleanup daemons or else you will have a content filtering
loop.

Limitations:

FILTER actions from smtpd access maps and header/body_checks
take precedence over filters specified with the main.cf content_filter
parameter.

If a message triggers more than one filter action, only
the last one takes effect.

The same content filter is applied to all the recipients
of a given message.