The DarkNet Project

Changes in version 1.4

Added ipf rules for NTP and DNS (thanks to Victor Wilson for the suggestion!).

Added ipf logging suggestions and examples.

Changes in version 1.2

Added additional routing suggestions.

Aesthetic changes.

Introduction

Tracking compromised machines can be difficult. Security solutions often
don't scale to the size of larger networks. Technologies such as IDS are
flawed, producing copious false positives. When solutions are scaled to fit
the larger providers, they often require considerable care and feeding, thus
taking time away from problem mitigation. There must be a better way!

Enter the Darknet! A Darknet is a portion of routed, allocated IP space
in which no active services or servers reside. These are "dark" because
there is, seemingly, nothing within these networks.

A Darknet does in fact include at least one server, designed as a packet
vacuum. This server gathers the packets and flows that enter the Darknet,
useful for real-time analysis or post-event network forensics.

Any packet that enters a Darknet is by its presence aberrant. No
legitimate packets should be sent to a Darknet. Such packets may have
arrived by mistake or misconfiguration, but the majority of such packets are
sent by malware. This malware, actively scanning for vulnerable devices,
will send packets into the Darknet, and this is exactly what we want.

Darknets have multiple uses. These can be used to host flow collectors,
backscatter detectors, packet sniffers, and IDS boxes. The elegance of the
Darknet is that it cuts down considerably on the false positives for any
device or technology.

The goals of the Darknet are simple - to increase awareness, and to ease
mitigation. With a Darknet in place, it is far easier to determine the
amount of naughty traffic on a network, as well as the sources of said
traffic.

Deploying a Darknet

Building a Darknet - Network

The first step in deploying a Darknet is to locate a suitable home. We
recommend first determining the amount of address space you plan to allocate
to the Darknet. At a minimum we recommend a /24, though smaller address
space - even up to a /32 - will work. You could even allocate a single /32
from each of several /24s to make the Darknet very difficult to detect. The
more address space you allocate to the Darknet, the greater your visibility
and statistical sampling.

Remember the rule of routing - the most specific prefix wins. This is a
great advantage for a Darknet. If you have a /16 for your enterprise, you
can route the entire /16 to your Darknet, with more specific routes in your
IGP carrying production traffic to the proper places. As with all things,
carefully ponder and test this configuration prior to deployment.

Do not use bogon address
space. Bogon space is avoided by many scanners and malware. Because bogon
space does not include vulnerable hosts, there is no return on the
investment for the malware. Use only allocated, routed space for the
Darknet.

Note: While this document uses IPv4 exclusively, an IPv6 Darknet
is also a viable solution. Future versions of this document may include IPv6
examples, depending largely on Rob's caffeine intake.

A /32, not within the allocated Darknet space, needs to be
assigned for both the management IP and the sniffer IP.

The next step is to set aside the physical and logical space for the
Darknet. You will need:

The Darknet IP prefix.

A sniffer IP NOT in the Darknet space.

A management IP NOT in the Darknet space.

A distinct VLAN or hub for the Darknet space.

We strongly recommend against putting the Darknet in the same collision
domain or VLAN as other subnets. The goal of the Darknet is to provide
credible data, thus it is critical to avoid "poisoning" the Darknet with
legitimate traffic.

We recommend against putting the Darknet IPs into your publicly visible
DNS. DNS RRs such as darknet-sniffer.your.net are just a
bit obvious. ;)

For our example, we'll use the following (these are BOGONS, so do not use them!):

An upstream router interface, or entire router if you have the luxury,
should be selected and configured for the Darknet. The router should be
configured to protect itself; the Secure IOS
Template or the Secure JunOS
Template can be of use here. Otherwise the router should permit all
packets to pass into the Darknet. The router can be configured to block all
outbound packets from the Darknet.

The router should be configured for SNMP, and at a minimum traffic
statistics should be collected from the interface facing the Darknet server.
This is done for two reasons:

New malware, be it worms or bots or other, can be quickly detected based solely on Darknet interface traffic statistics.

Any outbound traffic is a bad sign and should generate an alert.

The Witty worm is an example where our Darknets alerted us within minutes of the release of the worm. This set of graphics from three distinct Darknets makes it quite clear something was amiss:

There are myriad packages designed to collect router statistics, to
include MRTG. The router can also be
configured to export flows to a package of your choice. While this may be
redundant for a Darknet, it may also provide a seamless way to integrate
Darknet data into existing network monitoring infrastructure. We like the
following packages, though your mileage may vary:

Building a Darknet - Server

Everyone has a favorite server platform and operating system. We provide
these suggestions as a guideline, and you should feel free to modify them to
fit your environment. The goal of a Darknet is to increase awareness, not to
increase workload.

The server must have two NICs. One is for sniffing, and the other is for
management. Again it is imperative that we not contaminate our Darknet with
legitimate traffic, to include our own management traffic. In our example we
pick FreeBSD 4.8 and assign the two NICs thusly:

em0 - SNIFFER NIC - 172.16.18.2
fxp0 - MGMT NIC - 192.168.18.2

The default route for the box, if any, should point to the gateway on the
management subnet, e.g. the gateway upstream from the fxp0 interface in our
example. Do not point any routes out of the SNIFFER NIC. If you can go with
static routes and no default, all the better to protect the Darknet server.
Default routes are not always necessary.

route add -net <mgmt-subnet> 192.168.18.1

We also recommend adding a null route on the box for the allocated
Darknet prefix. This adds a bit of extra protection, and ensures that the
Darknet server can not answer packets destined for the Darknet prefix. Both
Solaris and FreeBSD support blackhole routes, which are implemented
thusly:

route add -net 10.18.18.0/24 127.0.0.1 -blackhole

Remember to update the boot files so that this route is implemented at
reboot.

We recommend a single CPU box, at least 1.8GHz for Intel, with at least
200GB of hard drive space. This depends entirely on the amount of data you
choose to log. In one of our Darknet pods, we can log a gigabyte per hour
from a single /16. Ultimately disk space and I/O are likely to be your
strictest bottlenecks. Ensure that you monitor the disk space utilization
and I/O throughput on the Darknet server.

We recommend using UTC (aka GMT) as the timezone on all of your Darknet
servers. A standard timezone will make cross-Darknet correlation a much
simpler task. This can be accomplished by copying the proper timezone file,
usually named "GMT," to /etc/localtime. Don't forget to synchronize time
with two or more NTP servers. In our example, the management servers
192.168.1.9 and 192.168.2.9 will be our NTP servers. There are several
features for NTP configuration, but at a minimum you want the following in
your ntp.conf file:

# Our NTP servers.
server 192.168.1.9
server 192.168.2.9

You likely want DNS running on your Darknet server. In our example we
will use the management servers 192.168.1.9 and 192.168.2.9 as our DNS name
servers (they also serve NTP). Be careful with DNS and sniffing! Be sure to
use the -n flag with tcpdump so that your Darknet server
doesn't attempt to resolve each IP in your sniffer output. Our
/etc/resolv.conf file should now include:

A discourse on how to build and implement each is beyond the scope of
this treatise. Each comes with ample build and installation documentation.
We will cover only certain tweaks here.

Argus and tcpdump should be configured to listen on the SNIFFER NIC only,
in our example em0. The remote management daemon for Argus should be bound
to the MGMT NIC only, e.g. fxp0, on port 2002 in our example. It should not
be bound to the SNIFFER NIC.

Argus should be run as a daemon, and ideally started at boot. Here is a
suggested template for your /etc/argus.conf file.

# /etc/argus.conf
# 01 MAY 2004 robt@cymru.com
#
# Run Argus as a daemon
ARGUS_DAEMON=yes
#
# Now select a _unique_ Argus ID for this Darknet server.
# Configure all Darknet servers with their own, unique
# Argus ID.
ARGUS_MONITOR_ID=5
#
# Select a port on which the Argus daemon will listen for
# remote management and queries. We've already chosen
# TCP 2002 as our port.
ARGUS_ACCESS_PORT=2002
#
# We only want to sniff the em0 port.
ARGUS_INTERFACE=em0
#
# Store the flows in an output file. Rotate this file
# regularly with a script.
ARGUS_OUTPUT_FILE=/var/log/argus/argus-id5-log
#
# Log the PID of the Argus daemon.
ARGUS_SET_PID=yes
#
# Set to promiscuous mode.
ARGUS_GO_PROMISCUOUS=yes
#
# Decision time. How often do you wish to receive reports
# on long-lived flows? A small number here is safe, because
# there should be _NO_ long lived flows in a Darknet. The
# Darknet isn't supposed to actually complete a connection
# request, after all. We'll set this to five seconds.
ARGUS_FLOW_STATUS_INTERVAL=5
#
# It's wise to regularly output the Argus daemon stats.
# These can be very helpful when debugging problems. We'll
# set this to 60 seconds.
ARGUS_MAR_STATUS_INTERVAL=60
#
# While we don't use these statistics regularly, they can
# be handy at times. Go ahead and collect them.
ARGUS_GENERATE_RESPONSE_TIME_DATA=yesARGUS_GENERATE_JITTER_DATA=yes
#
# MAC address information should be uninteresting, because
# all packets will have the source MAC of the upstream
# router.
ARGUS_GENERATE_MAC_DATA=no
#
# Capturing only headers can hinder analysis efforts. If
# you have the CPU and disk space, gather all the bits in
# each packet.
ARGUS_CAPTURE_DATA_LEN=1518
#
# Optimize the libpcap filters.
ARGUS_FILTER_OPTIMIZER=yes
#
# Do not filter the packets; grab them all.
ARGUS_FILTER=""
#
# The ability to run tcpdump under the Aegis of Argus is
# one of its most powerful features. This combines the
# flow logging with packet payload. If you have the CPU
# and disk capacity, enable this feature. Remember to
# rotate these logfiles!
ARGUS_PACKET_CAPTURE_FILE="/var/log/argus/argus-id5-tcpdump"
#

IP Filter should be configured to block everything in and out of the
SNIFFER NIC interface. In our example this is the em0 interface, IP address
172.16.18.2. You may wish to log any hits on the block rules so that
debugging and alerts are easy to accomplish; it's not a bad idea to do the
same on the pass rules as well. If you choose to log the rule hits, remember
to start ipmon with the proper flags.

Modify /etc/ipf.rules (/etc/opt/ipf/ipf.conf in Solaris)
thusly:

# Prevent any traffic from entering and exiting the em0
# SNIFFER NIC.
block in log quick on em0 from any to any
block out log quick on em0 from any to any
#

But won't this block the traffic we're attempting to log? No, both Argus
and tcpdump insert themselves beneath ipf, gathering all the packets and
flows before ipf blocks the traffic.

The Darknet server is now configured with SSH on TCP 22 and an Argus
remote flow collection port on TCP 2002. Both should be configured to bind
only to the MGMT NIC, not the SNIFFER NIC. These should also be protected
using ipf. Remember we are managing this device out of band from the Darknet
network.

# Lock down management access to the fxp0 MGMT NIC.
pass in quick on fxp0 proto tcp from <mgmt-subnet> to 192.168.18.2 port = 22 keep state
pass in quick on fxp0 proto tcp from <mgmt-subnet> to 192.168.18.2 port = 2002 keep state
block in log quick on fxp0 from any to any
#

Depending on the functions and management methods of your Darknet server,
you may wish to permit outbound connections from the MGMT NIC, permit
inbound ICMP, etc. Here is an example of a complete FreeBSD Darknet server
/etc/ipf.rules (Solaris /etc/opt/ipf/ipf.conf) file. In this
example the management subnets are 192.168.1.0/24 and
192.168.2.0/24. For guidelines on permissable ICMP messages, please
see our ICMP
Packet Filtering guide. It is wise to test new filter sets from an out
of band management port, e.g. the console, to avoid locking yourself out of
the Darknet server. Remember to modify this filter set to fit your
environment and adhere to applicable security policies.

# /etc/ipf.rules
# 01 MAY 2004 robt@cymru.com
#
# em0 172.16.18.2 SNIFFER NIC
# fxp0 192.168.18.2 MGMT NIC
#
# 192.168.1.0/24 ORD NOC
# 192.168.2.0/24 LHR NOC
#
# Prevent any traffic from entering and exiting the em0
# SNIFFER NIC.
block in log quick on em0 from any to any
block out log quick on em0 from any to any
#
# Lock down SSH, Argus, and SNMP access to the fxp0 MGMT NIC.
pass in quick on fxp0 proto tcp from 192.168.1.0/24 to 192.168.18.2 port = 22 keep state
pass in quick on fxp0 proto tcp from 192.168.2.0/24 to 192.168.18.2 port = 22 keep state
pass in quick on fxp0 proto tcp from 192.168.1.0/24 to 192.168.18.2 port = 2002 keep state
pass in quick on fxp0 proto tcp from 192.168.2.0/24 to 192.168.18.2 port = 2002 keep state
pass in quick on fxp0 proto udp from 192.168.1.0/24 to 192.168.18.2 port = 161 keep state
pass in quick on fxp0 proto udp from 192.168.2.0/24 to 192.168.18.2 port = 161 keep state
#
# Permit SSH outbound to the management subnets.
pass out quick on fxp0 proto tcp from 192.168.18.2 to 192.168.1.0/24 port = 22 keep state
pass out quick on fxp0 proto tcp from 192.168.18.2 to 192.168.2.0/24 port = 22 keep state
# Permit syslog outbound to the management subnets.
pass out quick on fxp0 proto udp from 192.168.18.2 to 192.168.1.0/24 port = 514 keep state
pass out quick on fxp0 proto udp from 192.168.18.2 to 192.168.2.0/24 port = 514 keep state
#
# Permit DNS outbound to the DNS name servers.
pass out quick on fxp0 proto udp from 192.168.18.2 to 192.168.1.9 port = 53 keep state
pass out quick on fxp0 proto tcp from 192.168.18.2 to 192.168.1.9 port = 53 keep state
pass out quick on fxp0 proto udp from 192.168.18.2 to 192.168.2.9 port = 53 keep state
pass out quick on fxp0 proto tcp from 192.168.18.2 to 192.168.2.9 port = 53 keep state
#
# Permit NTP outbound to the NTP servers.
pass out quick on fxp0 proto udp from 192.168.18.2 to 192.168.1.9 port = 123 keep state
pass out quick on fxp0 proto udp from 192.168.18.2 to 192.168.2.9 port = 123 keep state
#
# Permit some ICMP from the management subnets and the upstream router.
pass in quick on fxp0 proto icmp from 192.168.1.0/24 to 192.168.18.2 icmp-type echo
pass in quick on fxp0 proto icmp from 192.168.2.0/24 to 192.168.18.2 icmp-type echo
pass in quick on fxp0 proto icmp from 192.168.18.1 to 192.168.18.2 icmp-type echo
pass in quick on fxp0 proto icmp from 192.168.1.0/24 to 192.168.18.2 icmp-type echorep
pass in quick on fxp0 proto icmp from 192.168.2.0/24 to 192.168.18.2 icmp-type echorep
pass in quick on fxp0 proto icmp from 192.168.18.1 to 192.168.18.2 icmp-type echorep
pass in quick on fxp0 proto icmp from 192.168.1.0/24 to 192.168.18.2 icmp-type timex
pass in quick on fxp0 proto icmp from 192.168.2.0/24 to 192.168.18.2 icmp-type timex
pass in quick on fxp0 proto icmp from 192.168.1.0/24 to 192.168.18.2 icmp-type unreach
pass in quick on fxp0 proto icmp from 192.168.2.0/24 to 192.168.18.2 icmp-type unreach
#
# Permit some outbound ICMP for debugging purposes.
pass out quick on fxp0 proto icmp from 192.168.18.2 to 192.168.1.0/24 icmp-type echo
pass out quick on fxp0 proto icmp from 192.168.18.2 to 192.168.2.0/24 icmp-type echo
pass out quick on fxp0 proto icmp from 192.168.18.2 to 192.168.18.1 icmp-type echo
pass out quick on fxp0 proto icmp from 192.168.18.2 to 192.168.1.0/24 icmp-type echorep
pass out quick on fxp0 proto icmp from 192.168.18.2 to 192.168.2.0/24 icmp-type echorep
pass out quick on fxp0 proto icmp from 192.168.18.2 to 192.168.18.1 icmp-type echorep
#
# Block everything else.
block in log quick on fxp0 from any to any
block out log quick on fxp0 from any to any
#

Once you have the desired filter set in place, reload your IP Filter
firewall with:

ipf -f /etc/ipf.rules

Now fire up tcpdump on the em0 interface to see if the packets to our
Darknet prefix, 10.18.18.0/24, are arriving. After launching tcpdump, send a
few packets to sundry addresses within 10.18.18.0/24. If your chosen Darknet
prefix is accessible from the Internet, it won't take long before the
naughty packets arrive.

tcpdump -n -i em0

Next test that you can connect to the Argus port from your remote
management stations.

ra -zn -S 192.168.18.2 -P 2002

If you see packets with tcpdump and flows with Argus, congratulations!
You now have an operational Darknet. :)

A clever alternative is to use a host-based system. One suggestion is to
use Linux and iptables. Route a small prefix (e.g. a /25) to a Linux box
that is a "log and drop" iptables firewall. The data can be pulled from the
logs to produce results similar to the Darknet configuration detailed
here.

Using A Darknet

Managing a Darknet

Managing a Darknet shouldn't be an arduous task. There is the usual
health checking of both the router and the Darknet server. We recommend
using SNMP on both to monitor interface usage. It is unlikely to ever be
zero bps or pps. We also strongly recommend monitoring the processes on the
Darknet server, to include cron, syslog, Argus, and tcpdump. Keep a close
eye on the disk space on the Darknet server, as that can fill up rapidly
during periods of high network stress.

Tools such as rancid, netsnmp, and Big Brother can be of great use here. Any tool
should integrate with your existing network management and monitoring, of
course.

We recommend rotating the created logfiles, as they can become quite
large. tcpdump can be told to rotate the log file when it reaches a set size
with the -C flag. This example rotates the file and names it
"darkdumpN" (where N is an incrementing number) when it reaches 150MB in
size.

tcpdump -i em0 -n -w darkdump -C150

Scripts can also be used to rotate log files.

As with all things keep patches and code current and secure. Regularly
audit your Darknet. This is not the host you want compromised when a nasty
bit of malware visits your network.

Patrick Green of the OxCERT team at Oxford University, UK, has put
together a great paper on the practical use of a Darknet. Our thanks to Pat
for authoring the paper and permitting us to host it here. You will find the
paper at http://www.team-cymru.org/documents/dark-watcher.pdf.

Now What?

So you have a Darknet - now what do you do with it? There are a variety
of ways to peruse the collected data, and myriad uses for that data.

Perhaps you've heard that there is a bot that exploits Microsoft Windows
2000 open shares. This bot scans on TCP port 445, and you wish to know how
much of that traffic is reaching (or sourced within) your network. A quick
Argus query to your Darknet will help to answer that question. The query is
constructed using tcpdump pattern matching syntax. See the ra man page for more
examples.

Sure enough there is a fair bit of TCP 445 scanning occuring. The first
column of IPs are the sources; these can now be reported to the system
owners. What has compromised these hosts? Unclear, but what is clear is that
something is wrong - remember, no legitimate packets should ever enter a
Darknet.

Slammer was a worm that sent packets to UDP 1434. Argus and a Darknet can
spot these easily enough.

The data files can also be perused in batch mode with custom shell and
Perl scripts.

One use is to match up the flows with known malware signatures.

You can use this data to alert customers to problems within their
networks. For example, if you flag 300 Slammer scans from a customer
network, you can contact them with the source IP and UTC timestamp of the
scan. They can then take action to clean the reported hosts. If you're
reporting infected hosts to peers or other providers, you can run the list
through the Team Cymru WHOIS
server to make notification list easier to assemble.

You can use the data for statistical analysis. This is an excellent
method by which you can begin to quantify the amount of "trouble" on your
network. You can even create a simple "garbage meter" based on the traffic
entering your Darknets.

Customers and management will appreciate the proactive nature of
Darknets, meaning you can "spot bot" and other malware before it is used for
spam or DDoS.

You could readily host IDS boxes in your Darknet, thus matching on the
known fingerprints.

Best of all you can use your Darknet deployment and data to convince
others to do the same. Darknets aren't just for providers. They will fit
nicely in enterprise and (supposedly) disconnected networks.