netstat – network status

netstatThe status of the input and output through a network interface is assessed via the command netstat. This command can provide the complete information on how the network interface is performing, down to even socket level. Here is an example:

# netstatActive Internet connections (w/o servers)Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 prolin1:31027 prolin1:5500 TIME_WAIT tcp 4 0 prolin1l:1521 applin1:40205 ESTABLISHED tcp 0 0 prolin1l:1522 prolin1:39957 ESTABLISHED tcp 0 0 prolin1l:3938 prolin1:31017 TIME_WAITtcp 0 0 prolin1l:1521 prolin1:21545 ESTABLISHED… and so on …The above output goes on to show all the open sockets. In very simplistic terms, a socket is akin to a connection between two processes. [Please note: strictly speaking, “sockets” and “connections” are technically different. A socket could exist without a connection. However, a discussion on sockets and connections is beyond the scope of this article. Therefore I have merely presented the concept in an easy-to-understand manner.] Naturally, a connection has to have a source and a destination, called local and remote address. The end points could be on the same server; or on different servers.

In many cases, the programs connect to the same server. For instance, if two processes communicate among each other, the local and remote addresses will be the same, as you can see in the first line – the local and remote addresses are both the sever “prolin1”. However, the processes communicate over a port, which will be different. This port is shown next to the host name after the “:” (colon) mark. The user program sends the data to be sent across the socket to a queue and the receiver reads from a queue at the remote end. Here are the columns of the output:

The leftmost column “Proto” shows the type of the connection – tcp in this case. The column Recv-Q shows the bytes of data in the queue to be sent to the user program that established the connection. This value should be as close to 0 as possible. In busy servers this value will be more than 0 but shouldn’t be very high. A higher number may not mean much, unless you see a large number in Send-Q column, described below. The Send-Q column denotes the bytes in the queue to be sent to the remote program, i.e. the remote program has not yet acknowledged receiving it. This should be close to 0. A large number may indicate a network bottleneck. Local Address is source of the connection and the port number of the program. Foreign Address is the destination host and port number. In the first line, both the source and destination are on the same host: prolin1. The connection is simply waiting. The second line shows and established connection between port 1521 of proiln1 going to the port 40205 of the host applin1. It’s most likely an Oracle connection coming from the client applin1 to the database server prolin1. The Oracle listener on prolin1 runs on port 1521; so the port of the source is 1521. In this connection, the server is sending the requested data to the client. The column State shows the status of the connection. Here are some common values. ESTABLISHED – that the connection has been established. It does not mean that any data is flowing between the end points; merely that the end points have talked to each other. CLOSED – the connection has been closed, i.e. not used now. TIME_WAIT – the connection is being closed but there are still packets in the network that are being handled. CLOSE_WAIT – the remote end has shutdown and has asked to close the connection. Well, from the foreign and local addresses, especially from the port numbers, you can probably guess that the connections are Oracle related, but won’t it be nice to know that for sure? Of course. The -p option shows the process information as well:

RX-OK shows the number of packets successfully sent (for this interface) RX-ERR shows number of errors RX-DRP shows packets dropped and had to be re-sent (either successfully or not) RX-OVR shows packets overrun The next sets of columns (TX-OK, TX-ERR, etc.) show the corresponding stats for send data.

Flg column is a composite value of the property of the interface. Each letter indicates a specific property being present. Here is an explanation of the letters.

You can use the –interface (note: there are two hyphens, not one) option to display the same for a specific interface.

# netstat –interface=eth0 Kernel Interface tableIface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flgeth0 1500 0 277903459 0 0 0 170897632 0 0 0 BMsRUNeedless to say, the output is wide and is a little difficult to grasp at one shot. If you are comparing across interfaces, it makes sense to have a tabular output. If you want to examine the values in a more readable format, use the -e option to produce an extended output:

If you’d rather see the output showing IP addresses instead of host names, use the -n option.

The -s option shows the summary statistics of each protocol, rather than showing the details of each connection. This can be combined with the protocol specific flag. For instance -u shows the stats related to the UDP protocol.

One of the really useful options is the display of the routing table, the -r option.

# netstat -rKernel IP routing tableDestination Gateway Genmask Flags MSS Window irtt Iface10.20.191.0 * 255.255.255.128 U 0 0 0 bond0172.22.13.0 * 255.255.255.0 U 0 0 0 eth9169.254.0.0 * 255.255.0.0 U 0 0 0 eth9default 10.20.191.1 0.0.0.0 UG 0 0 0 bond0The second column of netstat output–Gateway–shows the gateway to which the routing entry points. If no gateway is used, an asterisk is printed instead. The third column–Genmask–shows the “generality” of the route, i.e., the network mask for this route. When given an IP address to find a suitable route for, the kernel steps through each of the routing table entries, taking the bitwise AND of the address and the netmask before comparing it to the target of the route.

The fourth column, Flags, displays the following flags that describe the route:

G means the route uses a gateway. U means the interface to be used is up (available). H means only a single host can be reached through the route. For example, this is the case for the loopback entry 127.0.0.1. D means this route is dynamically created. ! means the route is a reject route and data will be dropped. The next three columns show the MSS, Window, and irtt that will be applied to TCP connections established via this route.

MSS stands for Maximum Segment Size – the size of the largest datagram for transmission via this route. Window is the maximum amount of data the system will accept in a single burst from a remote host for this route. irtt stands for Initial Round Trip Time. It’s a little complicated to explain. Let me explain that separately. The TCP protocol has a built-in reliability check. If a data packet fails during transmission, it’s re-transmitted. The protocol keeps track of how long the takes for the data to reach the destination and acknowledgement to be received. If the acknowledgement does not come within that timeframe, the packet is retransmitted. The amount of time the protocol has to wait before re-transmitting is set for the interface once (which can be changed) and that value is known as initial round trip time. A value of 0 means the default value is used.

Finally, the last field displays the network interface that this route will use.