Managing and keeping tabs of network traffic on Linux

Is your Net connection slow? Ever wonder what the hell's going down the wire? Here are five free and effective tools to diagnose network traffic issues.

If you've been using Linux for a while, no doubt you're already amazed with the amount of software there is for you to use, for free. My own last count revealed in excess of three thousand applications ready to install and use. But how do we find the gems?

Look no further, because here they are. These are tools I use on a daily basis to both diagnose my own Internet connections and network performance in remote hosts. You can't claim to be a network management expert if you haven't used at least 5 of these tools, but you don't need to be an expert to use them either, because they're brutally easy to use.

The net essentials in our toolbox

Let's explore the first three tools in our toolset. They're powerful and easy enough for you to get an overview of what's going on with your Net connection.

KSysGuard: view aggregate network traffic

My favorite, in all aspects, is KSysGuard. With it, you can plot network usage for all your network interfaces (even though in 99% of all cases, there's only one network interface, and it's named eth0). Take a look at this screenshot:

It's not black magic. It's a sheet divided in rectangles, where you can drag and drop the sensors listed on the left to any of the empty rectangles on the right. When you place any of the sensors on the left, KSysGuard starts plotting.

Bonus points for KSysGuard: you can keep tabs on a remote machine via SSH. All you need to do is:

create an account on the remote machine,

install a small program named ksysguardd (which you can find after installing KSysGuard in the path /usr/bin/ksysguardd) on the remote machine,

set up passwordless SSH authentication for that account,

connect with KSysGuard to the remote machine using File -> Connect to machine...

The ksysguardd program is self-contained so you can be confident you won't be introducing security issues.

Here's an overview sheet plotted from a remote machine (in this case, this Web host):

Here's another, showing my MythTV PVR and backup machine:

Oh, did I mention KSysGuard can plot much, much more than just network traffic? I guess not, but the screenshots speak for themselves. Oh, and the screens are fully customizable, down to the colors used to plot the graphs. You can save the worksheets for later usage as well.

To install this tool, use your distribution's favorite package manager, and look for KSysGuard or the kdeadmin package.

In the next page, we'll explore the different uses of netstat.

(May I respectfully request, if you like this article, that you use one of the links right below to submit it to your favorite news site?)

netstat: dumb but it does the job

What if you want to find out which programs are communicating with hosts on the Internet? That's netstat's job.

Since netstat is a terminal program, I'm gonna open a terminal program to get the following screenshots. Do not worry -- you can follow the examples alright, because (despite what Windows zealots think) the terminal does not bite. Trust me: I was bitten by a Rottweiler once (quite the scare!), but so far not ever by the terminal. Swear to God!

(No, the terminal in Linux is not as stupid as MS-DOS. In fact, MS-DOS got nothin' on the Linux terminal. The Linux terminal can show you random fortune cookies -- try pulling fortunes out of MS-DOS, bitch! And unlike MS-DOS, you don't need to be a rocket scientist to type on the Linux console.)

You can see netstat (invoked with the tnp arguments -- TCP, show IP addresses, show programs) on that screenie, which comes up instantly after you've hit ENTER.

Sometimes it's useful to see host names instead of just IP addresses -- for that, you can use netstat -tp, removing the n argument. That will take a bit longer, but in the end will show something like this:

You can see I run a pretty tight machine -- by the way, hackers, don't bother because those services listening on all network interfaces are firewalled to the outside world.

The next page discusses mtr -- the rich man's traceroute.

mtr: the rich man's traceroute

I'm gonna stop for a while and thank my fabulous boss and friend Thomas Goirand for showing me this tool. Hi, Thomas! Up to the day he showed me this tool, I was stuck using the old traceroute and...

...I see I've abandoned my audience for a while there. OK, let's get back to business. traceroute is an old-school tool that attempts to find out the route Internet packets take from one machine to another. It sort of "pings" each hop in the way, displaying the time it took for each "pong" to turn around and "return to the mothership".

As you can see, it's very useful if you want to find out why a particular Internet host (that's technicalese for "computer") is not responding. Say I want to traceroute this host from home:

At three pings per hop (on each row) you can see my information takes eighteen hops from home to, well, this different home :-). Of course, if any of those hops is having a problem, traceroute will tell you -- and therein lies the value of traceroute, because it can tell you exactly where is the problem that makes a host inaccessible.

traceroute is nice and good, but slow. This trace took in excess of ten seconds. Also, I specified the -n argument which causes traceroute to show IP addresses instead of names -- otherwise it would have been really slow.

mtr beats the crap out of traceroute. It does the same thing, only in parallel (yes, even converting the IP addresses to meaningful names. I'm going to show you a screen capture, but a video (I'm too lazy for that) would do it more justice -- with mtr you see, in real time, the speed of each one of the hops in the way:

Moreover, you can keep it running and it will keep on running, updating its display every few times a second. Very useful to pinpoint the faulty spot on sporadic/temporary network outages!

Try it out. Install mtr using your distribution's package management utilities, then run mtr yahoo.com. Then try it on another host. Tip: if roundtrip times are more than two hundred milliseconds, you can forget about playing Unreal Tournament with that computer.

Oh, thanks again, Thomas!

Now, let's discuss flow analysis, and how to identify applications that are hogging your network.

Flow analysis: what's going on, and where

The tools we have seen so far are quite useful, but sometimes you want to point your finger at a network hog. And none of them are useful for that. What we'll do this time is inspect two (quite popular, I must confess) programs used to do flow analysis -- figuring out how traffic is apportioned, which program or network service is responsible, and where it goes.

iftop: network usage in real-time

I'm not gonna explain why the program iftop is named iftop -- I'm just gonna say iftop rocks. iftop. Damn, that's a lot of iftops on one sentence!

In short, iftop shows you how network traffic is being apportioned, breaking it down by host. This, for example, is a screenshot taken from my computer:

You can see three distinct flows in the three rows there, and the black bars depict the relative network usage for each flow (two bars per flow -- one for incoming and one for outgoing traffic). The last rows in the shot display the total network usage.

The two flows after the first are irrelevant -- we'll ignore them -- because they are broadcast traffic from either Avahi or Windows network browsing. As they say in the Wizard of Oz, do not pay attention to the man behind the curtain.

However, the first row is a lot more interesting. It shows a steady flow of data, in the vicinity of 1.6 megabits per second, from my "desktop" computer to my PVR computer. You see, my "desktop" computer is a Dell PowerEdge blade server, too loud to use as a desktop, and too cheap to have a sound card.

Since I'm a cheap and poor bastard, instead of bothering with an USB sound card for the PowerEdge, I just reuse one of the several sound cards in my PVR computer through the network and the magic of PulseAudio. Yes, I'm listening to music on my "desktop" computer, and the audio travels through my local network into my other computer, which relays the audio to my stereo, through my speakers, to my ears. You get the point.

Now here's a more meaningful screenshot, displaying network usage of my Internet connection (in this case, I typed the command iftop -i eth2 on the terminal window):

Now you can see a lot more flows -- why are the Messenger servers so intent on focusing in my computer? Well, I guess I now have homework to do, start investigating if anything's wrong.

Try iftop out. Use your Linux distribution's package manager to install iftop, then just run it on a terminal window (you may have to type sudo iftop to run it, because it requires administrative privileges). When you're bored, close it down. However, don't get surprised if you see a lot of traffic going in and out while downloading BitTorrent torrents.

Don't forget to read the iftop manual page as well (type man iftop on a terminal window). It's got lots of information for you to learn more, and a couple of other usage modes -- I like the one that shows me each protocol independently, and the other one that shows traffic in Bytes per second instead of bits per second. Very useful.

But there's a prettier tool.

Etherape: network usage in real-time, with graphical goodness

Remember iftop? Well, Etherape is like iftop, only cooler because it's a graphical program:

This is the very same network activity shown in the first iftop screenshot. Notice how the flow is now colored and has a distinct size based on which host is sending the traffic? Now here's a screenshot of Etherape capturing and displaying traffic on my Internet connection:

Note how it's much easier to discern network activity? Etherape colorizes traffic according to protocol, and each protocol gets a distinct color. You can use this neat tool to find out which kind of traffic is bogging your network down. Admittedly not very useful if you're using it on your own computer, but an absolute must if you are diagnosing traffic issues on an Internet gateway. You know, these pesky users downloading movies and music? They stand right out on Etherape.

The reason I prefer iftop to Etherape is simple, and related to my job: I can use iftop to diagnose remote hosts much more easily. When I run Etherape on another host, it runs slowly because my Internet connection isn't exactly great, and Etherape needs to send the pictures over the Internet. iftop's much quicker because it only sends characters. But you can't deny Etherape is prettier.

Now let's find out what exactly all these programs are talking about on the wire. You didn't know that was possible, did you?

Traffic inspection with Wireshark

As always, we're leaving the best for the last. Wireshark is different from all the tools you've seen so far. It's much harder to use, basically because it requires intimate knowledge of TCP/IP and many network concepts. It's also harder to use because there's no simple "overview" that you can derive from using Wireshark: it's down-to-the-wire details all the way.

Which makes it brutally useful to diagnose problems where identifying network chatter is essential to figuring out what is going wrong. Check this screenshot out to see what I mean:

Network chatter, as I was saying, is what Wireshark's all about:

1) Each row in the first pane displays an overview of each packet that Wireshark has captured.
2) The second pane provides a dissection of whatever packet is selected on the first pane, down to the bits and what each mean.
3) The third pane shows the actual packet contents in hexadecimal and textual views.

There's another view that warrants further attention. For most packets, you can right-click on any packet listed on the first pane, and choose an option titled Follow TCP stream:

Sorry about the blur you see on this screenshot, but I had to black out certain areas lest I be hacked like a steak. The point of the screenshot is to show you one of the essential features of Wireshark: network chat in real-time. In this shot, text colored in red was sent by the initiating host, while text colored in blue was sent by the responding host.

In short, Wireshark will capture network traffic from any host and to any host that crosses your Ethernet (or wireless) connections. It's one of those tools that "remove the magic factor" from computers, because it lets you see, in real-time, what the hell your computer (and others) is doing with your network connection.

As of now, I'm not sure if you perceive the looming danger with Wireshark. It's practically the reason all secure network protocols use encryption (OK, technically it's malicious Wireshark users sitting on network routers). Encrypted protocols appear mostly as gibberish to Wireshark. Unencrypted protocols, like HTTP and Telnet, do not. That's why I had to blur my screenshot.

However bad you think tools like this are widely available, you cannot deny their usefulness. Without it, I'd be out of a job -- keep in mind that my job does not include network penetration testing, yet I still find myriad uses for Wireshark.

I'm getting too political now. Just grab it and use it. If you don't, you'll never know what you were missing. If you do, you might learn a thing or two about network security you didn't know were there.

The wrap-up

OK, that's about everything you must know if you want to diagnose a network problem. I'm not going to show you the evil tools now (hey, Wireshark is borderline evil and outright illegal in Germany), but rest assured at some point I will show you how to use them as well.