Preventing Denial of Service Attacks

The Internet is no longer the cute and fluffy cloud it once was. Protecting
your servers, workstations, and networks can only go so far. Attacks that consume your available Internet-facing bandwidth or overpower your router's CPU
can still take you offline. This article will help you mitigate the effects of
such attacks, guiding you in what to do if you are attacked.

The techniques here apply equally well to FreeBSD 4.x and 5.x.

Different Types of Attacks

Denial of Service (DoS) attacks set out to remove a service from functional use by
its clients. Web servers will stop serving web pages, email servers will stop
accepting or delivering email, and routers will go dark, taking you off the
Internet all together.

Denial of a particular service will come in one of two forms:

Complete consumption of a resource such as bandwidth, memory, CPU, file handles, or any other finite asset.

Exploiting a weakness in the service to stop it functioning or causing the service to crash.

Over the last few years, attackers have refined their methods. As
developers make software more reliable and more resilient to DoS, the attack
vectors have changed to target hard-to-secure parts of a service. We'll
discuss the first type of attack and what we can do to protect our services
from it.

Make the Most of your Services

Protecting your services from attack is similar to tuning your services for
maximum performance. The greater the load you can handle, the more resilient
you are. Things change slightly when the attack alters the profile of your
service.

For example, if you have a web server tuned to transfer large files and the attack forces through a lot of small, short-lived transactions, you could find
you run out of network memory buffers very quickly. I would recommend starting
by reading the papers on tuning FreeBSD for different applications. The paper describes good ways to start tuning your servers. Also, the tuning(7)
man page is an excellent resource on performance improvements.

Analyzing and Blocking Denial of Service

The first step to protecting yourself from an attack is to understand the
nature of different types of attacks. As we said earlier, resource-consumption
attacks target your system in places that can cause bottlenecks. The most
popular targets are network bandwidth, system memory, network stack memory,
disk I/O, operating system limitations such as a limit on the number of open
file handles, and the CPU. These bottlenecks can be on your systems or in your
network hardware.

Attacks on Bandwidth

Attacks against your network bandwidth are difficult to defend. How you
deal with them depends heavily on your network topology and how helpful your
ISP is. Start by asking the following questions:

Is the attack against a single host, or multiple hosts?

Is the attacker hitting a small set of ports, or randomly hitting many ports?

Does the attack consists of protocols that would normally not be used with the attacked servers?

We are fortunate today that most attacks are simple in their nature. They
choose one or two styles of attack and at most a small number of IP addresses.
This makes sense — bandwidth is as hard for attackers to acquire as it is
for us to defend. If your Internet-peering bandwidth is not saturated, the
accepted approach is to block traffic to the attacked host(s) at your gateway.

It's a good idea to run tcpdump on the attacked servers if you can, to see
what kind of attack is taking place. Look for floods of very similar packets
— all TCP SYN, UDP, or ICMP. Look for packets all headed for a particular
port. If you find the number of source IP addresses is reasonably small, it may
be possible to block packets based on source address.

However, if the source
addresses are highly volatile in addressing, this can indicate spoofing or
forging. When this is the case, you may need to look for other similarities in
the attack such as packet size, window size, fragmentation, etc. If you have
the ability to block based on these less common criteria you may want to
investigate here further.

With modern, multi-gigabit networks, it is not unusual for an Internet
connection to have more bandwidth than the local LAN, so it may be possible for
you to block the attack at your Internet gateway. More often than not though,
this does not apply. Having your Internet bandwidth consumed can be tiring and
frustrating. This might be the right time to call your ISP, if they're willing
to work with you on these problems.

Before you make the call, try to analyze the attack. This will help your ISP
in selectively filtering the attack off your network. If filtering is
possible, you'll have one of two common options available: selectively
filtering out the attacking systems or dropping all packets to the attacked
servers. The latter is easier to manage and is more effective in the event that
the attack profile changes against those hosts.

If you run Border Gateway Protocol (BGP) at your Internet gateway to
announce your IP space to the Internet, you may have a third option, one that UUNet, C&W, XO, and many ISPs allow users to export routes as small as /32
with a special community string that causes their border routers to drop all
incoming data for the route. This is a highly effective method of dropping an
attack with the least damage to yourself and your ISP. Of course, this only
works well with a small number of hosts under attack and if your ISP offers
such functionality. Contact your ISP to find out. The obvious downside of this
is that the IP addresses you export in this fashion will lose all
connectivity to the Internet.

In general it is a good idea to keep your network clean; only allow the
traffic that your services need to operate. Allow TCP to ports 80 and 443 on
your servers and allow UDP to your game servers. Allow SSH connections only
from trusted hosts. All of these limit the options of the attackers when they
come to visit.

Attacks on Systems and Services

If your bandwidth is not saturated, the attack is most likely against your
systems and the services they host, rather than your entire network. Again, the
remedy depends on the nature of the attack. You may find that any one system
exposes multiple targeted bottlenecks. Attacks on systems and their services
generally fall into the following categories:

Network subsystem limitations (very high number of packets per second).

OS or application memory limitations (memory consumption).

Disk or CPU limitations (large numbers of valid requests).

System-targetting attacks are very frustrating, as they're hard to defend
against. FreeBSD does have some special defensive magic, however.

By default, each time your network card receives a packet, it generates an
interrupt to the CPU along its IRQ. The CPU will catch this and dedicate a
small amount of time to fetch this packet from the interface. Under normal
operations this can happen several thousand times per second — well
within the capabilities of even low end CPUs. It is quite likely with older
CPUs that you will start to see performance impacts between 25,000 to 50,000
packets per second. With packet sizes of 1,500 bytes, this works out to around
40Mbytes/sec to 75Mbytes/sec, quite a lot for most older CPUs to serve anyway.
Most 1Ghz systems will begin to feel pressure around 75,000 packets per second.
Two factors exacerbate the problem:

TCP SYN packets require full processing before the system can respond with
SYN ACK packets back to the source address. TCP and UDP packets to closed
ports as well as ICMP packets in general also need similar processing and
replies. While not as expensive as SYN processing, this still takes time and
consumes outbound bandwidth.

Packet size also plays an important factor. You can fit more small packets
in a particular amount of bandwidth than you can large packets. The more
packets you take in, the more CPU time you need to process them, no matter what
type of packets they are.

As we discussed previously, each interrupt request eats up some CPU time.
With enough IRQs generated, the CPU will have no time to do anything other than
serve the interrupts. Inbound packets stay unprocessed, applications receive
no CPU time, and your system is effectively dead in the water. This is known as
"Live-lock." Your system is still live, inasmuch as it has not crashed, but it
cannot perform any useful functions. Once packets stop coming in to the
interface, the CPU starts to process all of the backlogged packets it has
already accepted. This can take anything from a few minutes to several
hours.

There are several things you can do to prevent or mitigate the effects of a
high rate of packets without buying any hardware upgrades. All of these use
FreeBSD's sysctl(8)
command. Here are the settings you will need to place in
/etc/sysctl.conf:

net.inet.tcp.msl=7500

net.inet.tcp.msl defines the Maximum Segment Life. This is the
maximum amount of time to wait for an ACK in reply to a SYN-ACK or FIN-ACK, in
milliseconds. If the computer does not receive an ACK in this time, it
considers the segment lost and frees the network connection.

This has two implications. When you are trying to close a connection, if
the final ACK is lost or delayed, the socket will close more quickly. However, if a client is trying to open a connection to you and their ACK is delayed more
than 7,500 ms, the connection will not form. RFC 753 defines the MSL as 120
seconds (120,000 ms). However, this was written in 1979; timing issues have
changed slightly since then. Today, FreeBSD's default is 30,000 ms. This is
sufficient for most conditions, but for stronger DoS protection you can lower
this to 7,500 or less.

net.inet.tcp.blackhole=2

net.inet.tcp.blackhole defines what happens when the system
receives a TCP packet on a closed port. When set to 1, SYN packets
arriving on a closed port will be dropped without a RST packet being sent back.
When set to 2, all packets arriving on a closed port are dropped
without an RST being sent back. This saves CPU time, because packets don't need
as much processing, and outbound bandwidth, by not sending out
packets.

net.inet.udp.blackhole=1

net.inet.udp.blackhole resembles
net.inet.tcp.blackhole in its function. As the UDP protocol does
not have states like TCP, there is only one choice when it comes to dropping
UDP packets. When net.inet.udp.blackhole is 1, the
system will drop all UDP packets that arrive on a closed port.

net.inet.icmp.icmplim=50

The name net.inet.icmp.icmplim is somewhat misleading. This
controls the maximum number of ICMP "Unreachables" and also TCP RST packets to
return every second. It helps curb the effects of attacks that generate a lot
of reply packets.

kern.ipc.somaxconn=32768

kern.ipc.somaxconn limits the maximum number of concurrently
open sockets. The default here is just 128. If an attacker can flood you with a
sufficiently high number of SYN packets in a short enough period of time, he
can use up all of your possible network connections, successfully denying your
users access to the service.

You may find these settings to be either too aggressive or not aggressive
enough. Tune them until you receive satisfactory results.

Finally, if you are blessed enough to own one of the following network cards, you can enable a kernel feature call DEVICE_POLLING:

dc

em

fxp

nge

rl

sis

DEVICE_POLLING changed interrupt handling; with
DEVICE_POLLING, the kernel does not handle them at all! Instead,
at certain times, the CPU will poll the network card to pick up any packets that
are waiting for processing. This can significantly reduce the amount of CPU
time used in processing inbound traffic. This only works with the above cards,
as their drivers must support DEVICE_POLLING.

The FXP cards
generally work best with this feature, as their drivers and hardware are very
well developed. The hardware design and quality of RL cards is a lot lower
— without sufficient CPU (usually around 1Ghz), they have a hard time
achieving the full 100MB/s at all. Keep this in mind if you are looking for a
new network card.

You can learn more about DEVICE_POLLING at the author's home page. You can also find good installation and tuning instructions there, as well as statistics from comparative tests with DEVICE_POLLING enabled and disabled.

Tracking the Source of the Attack

Attacks can come from inside and outside your network. Obviously one is
easier to isolate than the other. Tracking the sources of attacks requires some
familiarity with packet-sniffing tools such as tcpdump, ngrep, and ethereal.
Unless you have spent several months carefully profiling your network traffic
and set up monitoring specifically to alert you of anomalies, the chances of
discovering you are under Denial of Service conditions before someone else does
are slim. More often than not, complaints such as "The Internet is slow" or "I can't get my email" will lead you to the truth. It is important to realize two things:

Attacks can come from inside and outside your network.

Not all service-denying events constitute a Denial of Service attack and not all Denial of Service attacks constitute a service-denying event.

What does this mean to you? It means that when you start to look for why
your Internet is slow or why people cannot download their email, remember that
the source of the problem could be from any machine on your network or the
Internet. If there's a denial, it may be accidental.

A good place to start is the point of bottleneck. This could be the CPU on
your HTTP proxy or your Internet gateway. If your bottleneck is a system
process such as a proxy server, examine the logs for this. Is a single system
or small number of systems making an unusually large number of requests or
using more resources than normal?

If your bottleneck is your Internet gateway
(which we assume is running FreeBSD), you can use the following command to view
the IP packets passing through your gateway:

router# tcpdump -n -i <interface> -c 100

This command will display a summary of the first 100 packets (-c
100) it sees on the given <interface> (-i
<interface>) and will not resolve the IP addresses to
host names (-n), which can take extra time and may itself fail if
you are having connectivity issues. An example output line will resemble:

Let us look at the first few parts of this output, which can be useful to us.

04:59:53.915324

This is the timestamp of when the packet was processed.

192.168.0.3.2327

This is the source IP address. The numbers after the last octet, 2327,
indicate the packet's source port number.

192.168.0.10.1214

This is the destination IP address. The numbers after the last octet, 1214,
indicate the destination port number.

S

This indicated the type of packet, in this case a SYN packet. See Daryl's TCP/IP Primer to learn about the process and life of a TCP connection and the types of packets you would see
here.

What you may see during an actual attack is hard to predict, as Denial of
Service attacks come in so many shapes and sizes. A typical attack involves
flooding a listening port on your server with SYN packets. The idea is to make
your system so busy processing the new connections that it cannot do anything
else. Here you may see a large number of SYN packets. Normally, you'll see a
balance of packets of all types.

References and Credits

This is a list of references I found when writing this article. Most of them
contain much more detail on their particular subject than I have provided here.
Hopefully they will be as useful to you as they were to me.

The FreeBSD handbook. A
great source of information about FreeBSD. Have a read here if you want to
learn something new about FreeBSD.

Google Groups. I used this
extensively to search USENET for articles others had posted in the past.
You should too!

HTMLHelp.com is made by the Web
Design Group to "promote the creation of non-browser specific, non-resolution
specific, creative, and informative sites that are accessible to all users
worldwide." It was valuable in the creation of this site.