General-Purpose Mail Filter

Appendix D Upgrading

The following sections describe procedures for upgrading
between the consecutive Mailfromd releases. The absense of a section for
a pair of versions x-y numbers means that no specific
actions are required for upgrading from x to y.

D.1 Upgrading from 8.5 to 8.6

New configure option --with-dbm allows you to select any
DBM flavor supported by GNU mailutils as the default DBM implementation
for mailfromd.

D.2 Upgrading from 8.2 to 8.3 (or 8.4)

Versions 8.3 and 8.4 differ only in required minimal version of
mailutils (3.3 and 3.4, correspondingly). Apart from that, the
following instructions apply to both versions.

In version 8.3 I abandoned the legacy DNS resolver and switched to
GNU adns. GNU ands is a standalone resolver library, which provides
a number of advanced features. It is included in most distributions. The
source code of the recent release is available from
http://www.chiark.greenend.org.uk/~ian/adns/adns.tar.gz.

This change brought a number of user-visible changes. In particular,
arbitrary limits on the sizes of the RRs are removed. Consequently,
the following configurations statements are withdrawn:

runtime.max-dns-reply-a

runtime.max-dns-reply-ptr

runtime.max-dns-reply-mx

max-match-mx

max-callout-mx

Secondly, the new resolver is less tolerant to deviations from the
standard. This means that remote DNS misconfigurations that would have
slipped unnoticed in previous versions, will be noticed by mailfromd
8.3. For example, a CNAME record pointing to another CNAME is treated
as an error.

A new command line option was added to mailfromd and
calloutd: --resolv-conf-file. This option instructs
the programs to read resolver settings from the supplied file, instead
of the default /etc/resolv.conf.

Another user-visible change is in handling of SPF checks. Previously,
results if SPF checks were cached in a database. This proved to cause
more problems than solutions and was removed in this version. As a
result, the following MFL global variables have been withdrawn:

spf_ttl

spf_cached

spf_database

spf_negative_ttl

D.3 Upgrading from 7.0 to 8.0

Version 8.0 is a major rewrite, that introduces a lot of new
concepts and features. Nevertheless, it is still able to run
the MFL scripts from version 7.0 without modifications.

These functions have been moved to the poll module, which must be
required prior to using any of them.

The message_header_count function.

This function takes an optional string argument, supplying the header
name. See message_header_count.

D.4 Upgrading from 6.0 to 7.0

The release 7.0 removes the features which were declared as obsolete
in 6.0 and introduces important new features, both syntactical, at the
MFL level, and operational.

Unless your filter used any deprecated features, it should work
correctly after upgrade to this version. It will, however, issue
warning messages regarding the deprecated features (e.g. the use
of ‘%’ in front of identifiers, as described below). To
fix these, follow the upgrade procedure described in upgrade procedure.

The removed features are:

Old-style functional notation

The use of functional operators

Implicit concatenations

#pragma option

#pragma database

The MFL syntax has changed: it is no longer necessary to
use ‘%’ in front of a variable to get its value. To reference a
variable, simply use its name, e.g.:

set x var + z

The old syntax is still supported, so the following statement
will also work:

set x %var + %z

It will, however, generate a warning message.

Of course, the use of ‘%’ to reference variables within a string
literal remains mandatory.

Several existing MFL functions have been improved. In
particular, it is worth noticing that the open function, when
opening a pipe to or from a command, provides a way to control where
the command’s standard error would go (see standard error).

The accept function (or action) issues a warning if its use would
cancel any modifications to the message applied by, e.g.,
header_add and similar functions. See Message modification queue, for a detailed discussion of this feature.

The most important change in mailfromd operation is that
the version 7.0 is able to run several servers (see conf-server).
Dedicated callout servers make it possible to run sender
verifications in background, using a set of long timeouts, as
prescribed by RFC 2822 (see SMTP Timeouts). This diminishes
the number of false positives, allows for coping with servers showing
large delays and also reduces the number of callouts performed for
such servers.

This release no longer includes the smap utility. It was
moved into a self-standing project, which by now provides much more
functionality and is way more flexible than this utility was. If you
are interested in using smap, visit
http://www.gnu.org.ua/software/smap, for a detailed
information, including pointers to file downloads.

D.5 Upgrading from 5.x to 6.0

The 6.0 release is aimed to fix several logical inconsistencies that
affected the previous versions. The most important one is that until
version 5.2, the filter script file contained both the actual filter
script, and the run-time configuration for mailfromd (in
form of ‘#pragma option’ and ‘#pragma database’ statements).
The new version separates run-time configuration from the
filter script by introducing a special configuration file
mailfromd.conf (see Mailfromd Configuration).

Consequently, the ‘#pragma option’ and ‘#pragma database’
statements become deprecated. Furthermore, the following deprecated
pragmas are removed: ‘#pragma option ehlo’, ‘#pragma option mailfrom’.
These pragmas became deprecated in version 4.0 (see 31x-400).

The second problem was that the default filter script file had
‘.rc’ suffix, which usually marks a configuration file, not
the source. In version 6.0 the script file is renamed to
mailfromd.mf. In the absence of this file, the legacy file
mailfromd.rc is recognized and parsed. This ensures backward
compatibility.

This release also fixes various inconsistencies and dubious features
in the MFL language.

The support for unquoted literals is discontinued. This feature
was marked as deprecated in version 3.0.

Mailfromd version ‘6.0’ will work with unchanged
scripts from ‘5.x’. When started, it will verbosely warn you
about any deprecated constructs that are used in your filter sources
and will create a script for upgrading them.

To upgrade your filter scripts, follow the steps below:

Run ‘mailfromd --lint’. You will see a list of warnings similar
to this:

This means that you use ‘#include’ where you should have used
‘require’. You will have to fix such warnings manually, as
suggested in the warning message.

If, for some reason, you cannot upgrade your scripts right now, you
may suppress deprecation warnings by setting the environment variable
MAILFROMD_DEPRECATION to ‘no’ before starting
mailfromd. Nonetheless, I recommend to upgrade as soon as
possible, because the deprecated features will be removed in version
‘6.1’.

D.6 Upgrading from 5.0 to 5.1

Upgrading from 5.0 to 5.1 does not require any changes in your
filter scripts. Notice, however, the following important points:

Starting from this release mailfromd supports Milter
protocol version 6, which is compatible with Sendmail 8.14.0 and
newer. While being backward compatible with earlier Sendmail
releases, it allows you to use the new ‘prog data’ handler
(see data). It also supports macro negotiation, a
feature that enables Mailfromd to ask MTA to export the
macros it needs for each particular handler. This means that if you
are using Sendmail 8.14.0 or higher (or Postfix 2.5 or higher), you no
longer need to worry about exporting macro names in sendmail.cf
file.

The same feature is also implemented on the server side, in
mtasim and pmult. Consequently, using
define-macros in pmult configuration file
is not strictly necessary. However, keep in mind that due to the
specifics of MeTA1, the number of symbols that may be
exported for each stage is limited (see pmult-macros).

The semantics of __preproc__ and __statedir__
built-in constant is slightly different from what it used to be in
5.0. These constants now refer to the current values of the
preprocessor command line and program state directory,
correspondingly. This should not affect your script, unless you
redefine the default values on run time. If your script needs to
access default values, use __defpreproc__ and
__defstatedir__, correspondingly (see Built-in constants).
The following example explains the difference between these:

The stack can be grown either by fixed size blocks, or
exponentially. Upper limit can be specified. See stacksize.

Milter ports can be specified using URL notation.

Removed deprecated features.

Support for some deprecated features has been withdrawn. These are:

Command line options --ehlo,
--postmaster-email, and --mailfrom.
These became deprecated in version 4.0. See 31x-400.

D.8 Upgrading from 4.3.x to 4.4

The deprecated --domain command line option has been
withdrawn. The short option -D now defines a preprocessor
symbol (see Preprocessor Options).

This version correctly handles name clashes between constants and
variables, which remained unnoticed in previous releases.
See variable--constant shadowing, for a detailed description of it.

To minimize chances of name clashes, all symbolic exception codes has
been renamed by prefixing them with the ‘e_’, thus, e.g. divzero
became e_divzero, etc. The ioerr exception code is renamed to
e_io. See status.mf, for a full list of the new exception codes.

For consistency, the following most often used codes are available without
the ‘e_’ prefix: success, not_found, failure, temp_failure. This
makes most existing user scripts suitable for use with version 4.4
without any modification. If your script refers to any exception
codes other than these four, you can still use it by defining a
preprocessor symbol OLD_EXCEPTION_CODES, for example:

$ mailfromd -DOLD_EXCEPTION_CODES

D.9 Upgrading from 4.2 to 4.3.x

Upgrading from 4.2 to 4.3 or 4.3.1 does not require any changes to
your configuration and scripts. The only notable change in these
versions is the following:

The asynchronous syslog implementation was reported to malfunction
on some systems (notably on Solaris), so this release does not enable
it by default. The previous meaning of the --enable-syslog-async
configuration option is also restored. Use this option in order to
enable asynchronous syslog feature. To set default syslog
implementation, use DEFAULT_SYSLOG_ASYNC configuration variable
(see syslog-async).

D.10 Upgrading from 4.1 to 4.2

Upgrading to this version does not require any special efforts. You
can use your configuration files and filter scripts without any
changes. The only difference worth noticing is that starting from this
version mailfromd is always compiled with asynchronous
syslog implementation. The --enable-syslog-async
configuration file option is still available, but its meaning has
changed: it sets the default syslog implementation to use
(see syslog-async). Thus, it can be used the same way it was in
previous versions. You can also select the syslog implementation at
run time, see –syslog-async option, for
more detailed information.

D.11 Upgrading from 4.0 to 4.1

Upgrading to this version does not require any special efforts. You
can use your configuration files and filter scripts without any changes.
Notice only the following major differences between 4.1 and 4.0:

Input files are preprocessed before compilation.
See Preprocessor, for more information.

There is a way to discern between a not-supplied optional
parameter, and a supplied one, having null value (see defined).

D.12 Upgrading from 3.1.x to 4.0

Starting from the version 4.0, MFL no longer uses the
predefined symbolic names for exception codes (previous versions used
the ‘&’ prefix to dereference them). Instead, it relies on
constants defined in the include file status.mfh
(see status.mf).

However, the script files from 3.1 series will still work, but
the following warning messages will be displayed:

Another important difference is that pragmatic options ‘ehlo’
and ‘mailfromd’ are now deprecated, as well as their command line
equivalents --ehlo and --domain. These options
became superfluous after the introduction of mailfrom_address
and ehlo_domain built-in variables. For compatibility with the
previous versions, they are still supported by mailfromd
4.0, but a warning message is issued if they are used:

D.13 Upgrading from 3.0.x to 3.1

The mailfromd binary no longer supports
--config-file (-c) option. To use an alternative
script file, give it as an argument, i.e. instead of:

$ mailfromd --config-file file.rc

write:

$ mailfromd file.rc

For backward compatibility, the old style invocation still works but
produces a warning message. However, if mailfromd
encounters the -c option it will print a diagnostic message
and exit immediately. This is because the semantics of this option
will change in the future releases.

If a variable is declared implicitly within a function, it is
created as automatic. This differs from the previous versions, where
all variables were global. It is a common practice to use global
variables to pass additional information between handlers (See HELO Domain, for an example of this approach). If your filter uses it,
make sure the variable is declared as global. For example, this code:

prog helo
do
# Save the host name for further use
set helohost $s
done

Figure D.1: Implicit declaration, old style

has to be rewritten as follows:

set helohost ""
prog helo
do
# Save the host name for further use
set helohost $s
done

Figure D.2: Implicit declaration, new style

Starting from version 3.1 the function dbmap takes an
optional third argument indicating whether or not to count the
terminating null character in key (see dbmap). If your startup
script contained any calls to dbmap, change them as follows:

This is necessary since the format of mailfromd databases
has changed in version 2.0: the key field now includes the trailing
‘NUL’ character, which is also reflected in its length. This
allows for empty (zero-length) keys. See Database Maintenance, for
more information about the database compaction.