After more than eight years since its first release in Phrack magazine, Fyodor has announced Nmap 4.00. Curious as usual, Federico Biancuzzi interviewed Fyodor on behalf of SecurityFocus to discuss the new port scanning engine, version detection improvements, and the new stack fingerprinting algorithm under work by the community.

Could you introduce yourself?

Fyodor:
I'm a long-time network security enthusiast with a particular
interest in full disclosure and the offensive side of security. I
have gained a lot from the security community over the years, and try
to contribute back by releasing free tools such as my Nmap Security Scanner and
publishing useful content on my websites, Insecure.Org and Seclists.Org. I am also an active
member of the Honeynet Project.
Writing has been a major recent focus of mine. Last year I
co-authored a technical security novel named Stealing the Network: How to Own a
Continent, and I'm almost finished with a network scanning book.
This is all on top of my active and varied social life. OK, I'm just
kidding about that last part. :)

You just released Nmap 4.00 after two years of work since 3.50.
What are the most exciting changes?

Fyodor:
Well, the Changelog
shows more than 230 improvements since that release, so it is hard to choose
just a few favorites. But some really do stand out. The port
scanning engine has been rewritten to be much faster and (after the
"diet Nmap" project) more memory efficient. The low-level packet
sending subsystem has changed dramatically as well. Nmap can now send
and route raw Ethernet frames rather than rely on the host's raw
sockets implementation. This is critical for Windows, since Microsoft
disabled raw sockets as of Windows XP SP2. And all platforms benefit
from the new ARP scanning and MAC address spoofing functionality that
this change allows.

Many Nmap users pick runtime interaction as
their favorite new feature. If you find yourself
staring at the screen wondering when Nmap will finish, just press
[enter] for an estimate. If you forgot to enable verbose mode, press
'v' to enable it. Or press 'V' to turn it off. Packet tracing and
debugging can be enabled or disabled on a whim as well.

Rewriting the port scanning engine produced a much faster version, and
thanks to the "diet Nmap" project it reduced memory consumption too. How
did you modify the code to get these advantages? And what type
of improvements can we expect?

Fyodor:
There are dozens of new algorithms and techniques in the new engine.
The most important is proably scanning multiple hosts at once (in
addition to scanning multiple ports on each host concurrently). Other
important performance changes include parallel DNS queries, "port scan
pings" that allows Nmap to detect network reliability and performance
even against heavily firewalled networks.

The best way to see the performance difference is to test Nmap 4.00
on your own network. All sorts of network characteristics play into
this, so blanket generalizations like "it is twice as fast" don't hold
water. And even external benchmarks may not be relevant to your
organization. But I know people want to see numbers, so I just did a
test scan of all 65K ports on www.insecure.org, seclists.org, and
scanme.nmap.org. I added the -T4 option for aggressive timing. The
tests were done from my home DSL line, which is limited to 128kbps
upsream. The scanning machine is running 64-bit Linux (AMD Athlon64
processor), which uses significantly more RAM than normal 32-bit
machines do. The memory consumption is based on the maximum 'VIRT'
value displayed by the top command, which includes Nmap code, data,
and shared library usage. I tested 3.50, 3.93 (which has the new
engine but preceeds the "diet Nmap" efforts), and 4.00. Also note
that these are heavily firewalled machines, which makes scanning them
relatively slow. With all that out of the way, here are the results:

As you can see, upgrading makes a huge difference. In some
situations, the speed increase may be greater, in other cases
worse. But Nmap 4.00 should never take longer than 3.50. If you find
a case where it does, let me know and I'll try to fix it.

How does the new ARP scan work?

Fyodor:
Well, Nmap has traditionally performed host discovery by spewing IP
packets throughout the target networks and listening for replies.
Obviously many firewalls block ICMP ping packets and other packet
types. Nmap gets around this by sending a user-specified combination
of IP packets in the hope that at least one will get through and
elicit a response. Nmap can send TCP SYN or ACK packets to a list of
ports, UDP packets to arbitrary ports, and ICMP netmask and
timestamp request queries.

But on local Ethernet networks, which is one of the most common Nmap usage
scenarios, there is a better solution. When Nmap tries to send a raw
IP packet such as an ICMP echo request, the operating system must
determine the destination hardware (MAC) address corresponding to the
target IP so that it can properly address the Ethernet frame. This is
often slow and problematic, since operating systems weren't written
with the expectation that they would need to do millions of ARP
requests against unavailable hosts in a short time period. Nmap now
takes over this role. And when it receives an ARP response back, it
knows the host is up so it doesn't even need to bother sending the IP
packets. The net effect is that scanning your local network is now
much faster and more accurate. Nmap automatically detects when
conditions are proper for doing this, so you don't even need to
remember an extra flag.

Can you tell us more about the version detection improvements?

Fyodor:
I'd love to, as that is an area which has improved greatly. I also
believe it is under-appreciated because many old time Nmap users
still don't realize it exists. Version detection allows you to
actually determine what service protocol (and often application name
and version number) is running on an open port, rather than simply
guessing based on the port number.

The biggest change is that the database has tripled in size to
3,153 signatures for 381 service protocols. We have most of the
common protocols (http, ftp, smtp, etc.) covered, as well as hundreds
of obscure ones from abc, acap, afp, and afs to zebedee, zebra, and
zenimaging. There is also a new --version-intensity option which
lets you specify how hard Nmap tries to interrogate the port. A low
number leads to a faster scan, while higher numbers are a bit more
likely to identify the service.

Another new feature is the exclude directive. One problem people
occasionally reported with version detection is that it would make
their printers spew out page after page of Nmap service probes,
including binary DNS and X11 requests that came out as gibberish. It
turns out that many printers listen on port 9100 and simply print out
whatever garbage is sent there. This is a dumb (printer) feature, but
the new exclude directive works around this by making Nmap skip that
port (only for version detection purposes) by default. You can
override that with the new --allports option.

Two more new features are the fallback directive and detailed
softmatches. These make version detection faster and more accurate,
without requires to even care about the implementation details. But
readers who are curious anyway can learn more from our new version detection
paper.
Many of these version detection changes were done by Doug Hoyte, Dirk
Mueller, Lionel Cons, Martin Macok, and Bo Jiang. I would also like
to thank the literally thousands of users who submitted service
fingerprints to the CGI that Nmap provides when it fails to identify a
responsive service.