Kernel Patch

Use the appropriate kernel patch from the "Layer 7 patches" package
to patch[1] the kernel (read the README
in the package to determine which patch to use). Set up your kernel as
you would otherwise. Now enable the following options (these are
correct for Linux 2.6.21.1, but they tend to move around a lot, so you may
have to go hunting if you have a different kernel version):

Optional but highly recommended: Lots of other Netfilter options,
notably "FTP support" and
other matches. If you don't know what you're doing, go ahead and enable
all of them.

Warning: Some users have reported kernel crashes
when they using SMP with
l7-filter. (Some have also reported that their SMP systems run fine.)
If you have a multi-CPU machine, test carefully before putting it into
production with l7-filter.

Actually doing stuff

There are three things you may be interested in
doing: (1) blocking certain protocols (2) controlling bandwidth use (3)
accounting. We cover each of these cases below.

First, a reminder: Just because you're using l7-filter, you don't
need to do all of your packet classification using it.
It's likely that what you want to accomplish can be at least partially
done with less demanding classifiers, such as port matching. For
instance, you can probably assume that traffic on TCP port 80 that isn't matched by any P2P patterns is HTTP; you don't need to actually use the HTTP
pattern.

l7-filter uses the standard iptables extension syntax. (If you are
not familiar with this, it's time to read the documentation at netfilter.org or at least "man
iptables".)

The only trick is that, in order to do its classification, l7-filter
must be able to see all of the relevant traffic. It
only sees packets if they go through an l7-filter rule.
One way of ensuring this is to use the POSTROUTING chain of
the mangle table:

iptables -t mangle -A POSTROUTING -m layer7 --l7proto [etc.]

See this packet flow diagram for
details. In some cases, l7-filter can sucessfully match even if it can
only see one side of the connection, but in general, this won't
work.

If you are using a version of l7-filter earlier than 2.7, you must
manually load the ip_conntrack module kernel for l7-filter to work.
Newer versions do this automatically.

1. Blocking

Don't. Here's why:

l7-filter matching isn't foolproof: there may be both false
positives (one protocol can look like another) and false negatives
(applications can do obscure things that we didn't count on). Patterns
that are known to regularly generate false positives are marked
"overmatching" on the protocols page, but others
may also do so occasionally.

Almost every type of Internet traffic has legitimate uses. For
instance, P2P protocols, while widely used to violate copyright, are
also an efficient way to distribute open source software and legally
free music.

Programs can respond to being blocked by port-hopping, switching
between TCP and UDP, opening a new connection for every trivial
operation, using encryption, or employing other evasion tactics. Trying
to block such protocols has consequences on two levels:

In the case of port/protocol-hopping, you make it harder for
yourself to identify protocols that already act this way.

You encourage programmers to include these "features" in new
programs, making it harder for everyone in the future. For example: In
early 2006, Bittorrent started moving towards end-to-end
encryption because many networks were either blocking it or severely
restricting its bandwidth.

l7-filter patterns are not generally designed with blocking in mind.
We consider a protocol to be well identified if the identification is
useful for controlling its bandwidth. This means, for instance, that
for P2P applications, we do not focus on catching connections that are
not downloads.

Blocking with l7-filter provides no security, since
any reasonably determined person can easily circumvent it.

Instead of dropping packets you don't like, we recommend using Linux
QoS to restrict their bandwidth
usage. If you insist on using l7-filter to drop packets, make sure you
have investigated other options first, such as the features of your HTTP
proxy (useful for worms).

2. Bandwidth Restriction

To control the bandwidth that a protocol uses, you can use Netfilter
to "mark" the packets and QoS to filter on that mark. To mark:

Did you understand that last command? You can try reading The Linux Advanced Routing and Traffic Control
HOWTO for enlightenment. You should do this so that you have some
idea what you're doing, but unfortunately, tc is incredibly
obtuse and you're likely to wish you just had a canned script. Well, we
can help:

These may need to be modified if your setup is significantly
different than mine, but it should provide a much better starting point
than most other things you are likely to find.

Be prudent when choosing the amount of bandwidth you allow each
protocol. Restricting a protocol to an unusably low bandwidth can have
similar consequences to blocking it.

3. Accouting

If you just want to keep track of what's in use on your network,
simply use the above command without any -j option. For example:

iptables -t mangle -A POSTROUTING -m layer7 --l7proto imap

You can then get statistics by using iptables -L. (See
"man iptables" for details.)

More Information

Dealing with FTP, IRC, etc.

Some protocols open child connections to transfer data. FTP is the
most familiar example. If you have loaded the
ip_conntrack_ftp or nf_conntrack_ftp kernel
module, l7-filter will classify FTP and all its child connections as
FTP. The same goes for IRC/IRC-DCC, etc.

If you wish to classify the children differently, use the standard
iptables "helper" match. You can use "-m --helper ftp" to
match ftp child connections. Of course, once you've done this, it's
silly to involve l7-filter, at least for the children.

The "unset" and "unknown" matches

l7-filter marks unmatched connections that it is still trying to
match as "unset". The first few packets of all TCP connections as well
as those of some UDP connections will match this. Similarly, l7-filter
marks connections that it has given up trying to match as "unknown".
These are matched just like normal protocols:

Upgrading the protocol definitions

The protocol definitions are simple text files with a format
described in the Pattern-HOWTO. They can be
updated as a package or individually.

If you update the protocol definitions, you need to clear the
relevant iptables rules and re-enter them. This is because the pattern
files are only read by iptables, not directly by the kernel.

Other things to know

By default, l7-filter looks at the first 10 packets or 2kB, whichever is smaller. These limits are
somewhat conservative. It is well known that some HTTP connections
(those that involve large cookies), for instance, need more packets to
be matched.

You can alter the number of packets at any time through
/proc/net/layer7_numpackets. (i.e. "echo 16 >
/proc/net/layer7_numpackets".)

In l7-filter versions 2.0 and forward, you can alter the number of
bytes at module load time: "modprobe xt_layer7
maxdatalen=N" (ipt_layer7 in old versions), where
N is in bytes. This should be used cautiously, since
performance may decrease drastically with larger data
sizes. To prevent you from accidentally bringing down your network,
there is an artificial limit of 65536 imposed. If you're sure you know
what you're doing, you can remove this limit by editing ipt_layer7.c or
xt_layer7.c in the kernel source.

It's possible (although rare) for a connection to be matchable by
more than one pattern. The patterns are tested in the order you
specified with iptables. After a match is made, l7-filter does not
continue testing that connection, so changing the order of your rules
may change what happens.

Sometimes important messages go only to the system log, not the
terminal you are working at. Such messages include notifications that
regular expressions failed to compile and various things that
tc generates. A useful command is "tail -f /var/log/messages".