Nmap Network Scanning

Nmap API

Chapter 9. Nmap Scripting Engine

Nmap API

NSE scripts have access to several Nmap facilities for writing
flexible and elegant scripts. The API provides target host
details such as port states and version detection results. It
also offers an interface to the Nsock
library
for efficient network I/O.

Information Passed to a Script

An effective Nmap scripting engine requires more than just a
Lua interpreter. Users need easy access to the information
Nmap has learned about the target hosts. This data is passed
as arguments to the NSE script's
action method.
The arguments, host and
port, are Lua tables which contain
information on the target against which the script is
executed. If a script matched a hostrule, it gets only the
host table, and if it matched a portrule it
gets both host and port.
The following list describes each variable in these two tables.

host

This table is passed as a parameter to the rule and action
functions. It contains information on the operating system run by
the host (if the -O switch was supplied), the
IP address and the host name of the scanned target.

host.os

An array of OS match tables. An OS match consists of a
human-readable name and an array of OS classes. Each OS
class consists of a vendor, OS family, OS generation,
device type, and an array of
CPE entries
for the class. (See the section called “Decoding the Reference Fingerprint Format”
for a description of OS match fields.) Fields may be
nil if they are not defined. The
host.os table has this overall
structure:

Only entries corresponding to perfect OS matches are put
in the host.os table. If Nmap was run
without the -O option, then
host.os is nil.

host.ip

Contains a string representation of the IP address of the
target host. If the scan was run against a host name and its
DNS lookup returned more than one IP addresses, then the
same IP address is used as the one chosen for the scan.

host.name

Contains the reverse DNS entry of the scanned target host
represented as a string. If the host has no reverse DNS entry,
the value of the field is an empty string.

host.targetname

Contains the name of the host as specified on the command line.
If the target given on the command line contains a netmask or is an IP
address the value of the field is nil.

host.reason

Contains a string representation of the reason why the target host is in
its current state. The reason is given by the type of the packet that
determined the state. For example, an echo-reply from
an alive host.

host.reason_ttl

Contains the TTL value of the response packet, that was used to determine
the status of the target host, when it arrived. This response packet is the
packet that is also used to set host.reason.

host.directly_connected

A Boolean value indicating whether or not the target host is
directly connected to (i.e. on the same network segment as) the host running Nmap.

host.mac_addr

MAC address
of the destination host (six-byte-long binary
string) if available, otherwise nil. The MAC address is generally only available for hosts directly connected on a LAN and only if Nmap is doing a raw packet scan such as SYN scan.

host.mac_addr_next_hop

MAC address
of the first hop in the route to the host, or
nil if not available.

host.mac_addr_src

Our own MAC address, which was used to connect to the
host (either our network card's, or (with
--spoof-mac)
the spoofed address).

host.interface

A string containing the interface name
(dnet-style)
through
which packets to the host are sent.

host.interface_mtu

The MTU (maximum transmission unit) for host.interface,
or 0 if not known.

host.bin_ip

The target host's IP address as a 4-byte (IPv4) or 16-byte (IPv6) string.

This table contains Nmap's timing data for the host (see
the section called “Round Trip Time Estimation”). Its keys are srtt (smoothed
round trip time), rttvar (round trip time variance), and timeout
(the probe timeout), all given in floating-point seconds.

host.traceroute

This is an array of traceroute hops, present when the
--traceroute option was used. Each entry is a
host table with fields name,
ip and times.srtt (round
trip time). The TTL for an entry is implicit given its position
in the table. An empty table represents a timed-out hop.

port

The port table is passed to an NSE service script (i.e. only those with a portrule rather than a hostrule) in the same
fashion as the host table. It contains information about the port
against which the script is running. While this table is not passed to host scripts, port states on the target can still be requested from Nmap
using the nmap.get_port_state() and nmap.get_ports() calls.

port.number

Contains the port number of the target port.

port.protocol

Defines the protocol of the target port. Valid values are
"tcp" and "udp".

port.service

Contains a string representation of the service running on
port.number as detected by the Nmap service
detection. If the port.version field is
nil, Nmap has guessed the service based
on the port number. Otherwise version detection was able to determine the listening service and this field is equal to
port.version.name.

port.reason

Contains a string representation of the reason why the target port is in
its current state (given by port.state). The reason is
given by the type of the packet that determined the state. For example, a
RST packet from a closed port or
SYN-ACK from an open port.

port.reason_ttl

Contains the TTL value of the response packet, that was used to determine
the status of the target port, when it arrived. This response packet is the
packet that is also used to set port.reason.

port.version

This entry is a table which contains information
retrieved by the Nmap version scanning engine. Some
of the values (such as service name, service type
confidence, and the RPC-related values) may be retrieved by
Nmap even if a version scan was not performed. Values
which were not determined default to
nil. The meaning of each value is given in the following table:

Table 9.1. port.version values

Name

Description

name

Contains the service name Nmap decided on for the port.

name_confidence

Evaluates how confident Nmap is about the accuracy of name, from 1 (least confident) to 10.

List of CPE codes for the detected service. As described in the
official CPE specification these strings
all start with the cpe:/ prefix.

port.state

Contains information on the state of the port.
Service scripts are only run against ports in the
open or
open|filtered states, so
port.state generally contains one
of those values. Other values might appear if the port
table is a result of the
get_port_state or get_ports
functions. You can adjust the port state using the
nmap.set_port_state() call. This is
normally done when an open|filtered
port is determined to be open.

Network I/O API

To allow for efficient and parallelizable network I/O, NSE
provides an interface to Nsock, the Nmap socket library. The
smart callback mechanism Nsock uses is fully transparent to
NSE scripts. The main benefit of NSE's sockets is that they
never block on I/O operations, allowing many scripts to be run in parallel.
The I/O parallelism is fully transparent to authors of NSE scripts.
In NSE you can either program as if you were using a single
non-blocking socket or you can program as if your connection is
blocking. Even blocking I/O calls return once a
specified timeout has been exceeded. Two flavors of Network I/O are
supported: connect-style and raw packet.

Connect-style network I/O

This part of the network API should be suitable for most
classical network uses: Users create a socket, connect it to a
remote address, send and receive data and finally close the socket.
Everything up to the Transport layer (which is either TCP, UDP or
SSL) is handled by the library.

An NSE socket is created by calling
nmap.new_socket, which returns a socket object.
The socket object supports the usual connect,
send, receive, and
close methods. Additionally the functions
receive_bytes,
receive_lines, and
receive_buf allow greater control
over data reception.
Example 9.3
shows the use of connect-style network operations. The
try function is used for error handling, as described in
the section called “Exception Handling”.

Raw packet network I/O

For those cases where the connection-oriented approach is too high-level,
NSE provides script developers with the
option of raw packet network I/O.

Raw packet reception is handled through a
Libpcap
wrapper inside the Nsock
library.
The steps are to open a capture device, register listeners
with the device, and then process packets as they are
received.

The pcap_open method creates a handle for raw socket reads from an
ordinary socket object. This method takes a
callback function, which computes a packet hash from
a packet (including its headers). This hash can return any
binary string, which is later compared to the strings
registered with the pcap_register
function. The packet hash callback will normally extract some
portion of the packet, such as its source address.

The pcap reader is instructed to listen for certain
packets using the pcap_register function.
The function takes a binary string which is compared against
the hash value of every packet received. Those packets whose
hashes match any registered strings will be returned by the
pcap_receive method. Register the empty
string to receive all packets.

A script receives all packets for which a listener has
been registered by calling the
pcap_receive method. The method blocks
until a packet is received or a timeout occurs.

The more general the packet hash computing function is
kept, the more scripts may receive the packet and proceed with
their execution. To handle packet capture inside your
script you first have to create a socket with
nmap.new_socket and later close the socket
with socket_object:close—just like
with the connection-based network I/O.

While receiving packets is important, sending them is certainly
a key feature as well. To accomplish this, NSE provides access to
sending at the IP and Ethernet layers. Raw packet writes do not use
the same socket object as raw packet reads, so the nmap.new_dnet
function is called to create the required object for sending. After
this, a raw socket or Ethernet interface handle can be opened for use.

Once the dnet object is created, the function ip_open
can be called to initialize the object for IP sending. ip_send
sends the actual raw packet, which must start with the IP header.
The dnet object places no restrictions on which IP hosts may be sent
to, so the same object may be used to send to many different hosts
while it is open. To close the raw socket, call ip_close.

For sending at a lower level than IP, NSE provides functions for
writing Ethernet frames. ethernet_open initializes
the dnet object for sending by opening an Ethernet interface. The raw
frame is sent with ethernet_send. To close the
handle, call ethernet_close.

Sometimes the easiest ways to understand complex APIs is by
example. The
ipidseq
script included with
Nmap uses raw IP packets to test hosts for suitability for Nmap's
Idle Scan (-sI). The
sniffer-detect
script also included with Nmap uses raw Ethernet frames in an attempt
to detect promiscuous-mode machines on the network (those running
sniffers).

Structured and Unstructured Output

NSE scripts should usually return a table representing their
output, one that is nicely organized and has thoughtfully chosen
keys. Such a table will be automatically formatted for screen
output and will be stored as nested elements in XML output.
Having XML output broken down logically into keys and values
makes it easier for other tools to make use of script output.
It is possible for a script to return only a string, but doing
so is deprecated. In the past, scripts could only return a
string, and their output was simply copied to the XML as a blob
of text–this is now known as “unstructured
output”.

Suppose a script called user-list returns a
table as shown in this code sample. The following paragraphs
show how it appears in normal and XML output.

A Lua table is converted to a string for normal output. The way
this works is: each nested table gets a new level of
indentation. Table entries with string keys are preceded by the
key and a colon; entries with integer keys simply appear in
order.
Unlike normal Lua tables, which are unordered, a table that
comes from stdnse.output_table will keep its keys in
the order they were inserted.
Example 9.4, “Automatic formatting of NSE structured output” shows how the
example table appears in normal output.

The XML representation of a Lua table is constructed as follows.
Nested table become table elements. Entries of
tables that are not themselves tables become elem
elements. Entries (whether table or
elem) with string keys get a key
attribute (e.g.
<elem key="username">foo</elem>);
entries with integer keys have no key element and
their key is implicit in the order in which they appear.

In addition to the above, whatever normal output the script
produces (even if automatically generated) is copied to the
output attribute of the script
element. Newlines and other special characters will be encoded
as XML character entities, for example &#xa;.
Example 9.5, “NSE structured output in XML” shows how the example
table appears in XML.

Some scripts need more control their normal output. This is the
case, for example, with scripts that need to display complex
tables. For complete control over the output, these scripts may
do either of these things:

return a string as second return value, or

set the __tostring metamethod on the
returned table.

The resulting string will be used in normal output, and the
table will be used in XML as usual. The formatted string may
contain newline characters to appear as multiple lines.

If the above code example were modified in this way to return a
formatted string,

There are conventions regarding the formatting of certain kinds
of data in structured output. Users of NSE output benefit by
being able to assume that some kinds of data, for instance dates
and times, are formatted the same way, even in different
scripts.

Network addresses, for example IPv4, IPv6, and MAC, are
represented as strings.

Long hexadecimal strings such as public key fingerprints should
be written using lower-case alphabetical characters and without
separators such as colons.

Dates and times are formatted according to
RFC
3339.
If the time zone offset is known, they should appear like these
examples:

2012-09-07T23:37:42+00:00
2012-09-07T23:37:42+02:00

If the time zone offset is not known (representing some
unspecified local time), leave off the offset part:

2012-09-07T23:37:42

The standard library function
stdnse.format_timestamp code exists to format times
for structured output. It takes an optional time zone offset in
seconds and automatically shifts the date to be correct within
that offset.

stdnse.format_timestamp(os.time(), 0) --> "2012-09-07T23:37:42+00:00"

Exception Handling

NSE provides an exception handling mechanism which is not present in
the base Lua language. It is tailored
specifically for network I/O operations, and
follows a functional programming paradigm rather than an
object-oriented one. The nmap.new_try API method is used to
create an exception handler. This method returns a function which takes a variable
number of arguments that are assumed to be the return values of
another function. If an exception is detected in the return
values (the first return value is false),
then the script execution is aborted and no
output is produced. Optionally, you can pass a function to
new_try which will be called
if an exception is caught. The function would generally perform any required cleanup operations.

Example 9.6 shows cleanup
exception handling at work. A new function named
catch is defined to simply close the
newly created socket in case of an error. It is then used
to protect connection and communication attempts on that
socket. If no catch function is specified, execution of the
script aborts without further ado—open sockets will
remain open until the next run of Lua's garbage
collector. If the verbosity level is at least one or if the
scan is performed in debugging mode, a description of the
uncaught error condition is printed on standard output.
Note that it is currently not easily possible to group
several statements in one try block.

Writing a function which is treated properly by the
try/catch mechanism is straightforward. The function should
return multiple values. The first value should be a Boolean
which is true upon successful completion of the function and
false (or nil) otherwise. If the function completed successfully, the try
construct consumes the indicator value and returns the
remaining values. If the function failed then the second
returned value must be a string describing the error
condition. Note that if the value is not
nil or false it is
treated as true so you can return your
value in the normal case and return nil, <error description>
if an error occurs.

The Registry

Scripts can share information by storing values in a
register, which is a special table that can be
accessed by all scripts. There is a global registry with the name
nmap.registry, shared by all scripts. Each host
additionally has its own registry called
host.registry, where host is the
host table passed to a script.
Information in the registries is not stored between Nmap
executions.

The global registry persists throughout an entire scan session.
Scripts can use it, for example, to store values that will later be
displayed by a postrule script. The per-host registries, on the other
hand, only exist while a host is being scanned. They can be used to send
information from one script to another one that runs against the same
host. When possible, use the per-host registry; this not only saves you
from having to make key names unique across hosts, but also allows the
memory used by the registry to be reclaimed when it is no longer
needed.

Here are examples of using both registries:

The portrule of the ssh-hostkey script collects SSH key fingerprints
and stores them in the global nmap.registry so they
can be printed later by the postrule.

The ssl-cert script collects SSL certificates and
stores them in the per-host registry so that the
ssl-google-cert-catalog script can use them without
having to make another connection to the server.

Because every script can write to the global registry table, it is
important to make the keys you use unique, to avoid overwriting the keys
of other scripts (or the same script running in parallel).

Scripts that use the results of another script must declare it using
the dependencies variable to make sure that the earlier
script runs first.