If you look at the Program Name column on the right, you’ll see that
apparently I have cupsd (printing), postgres, and redis servers running
on my machine, as well as some other stuff that I redacted. I actually
have no idea why I had redis installed so uh yeah I uninstalled it.

I use netstat pretty often when I’m trying to debug “omg why is this
thing not running IT IS SUPPOSED TO BE RUNNING”. netstat tells me the
truth about whether it is running.

dstat

Want to know how much data is actually being written to your physical hard
drive right this second? YEAH YOU DO. dstat knows that. It prints a row every
second with stats for that second. I love dstat because it’s so simple.

lsof

lsof tells you which files every process has open right now! That’s all!
It is awesome the same way dstat and netstat are awesome – you want to
know what files are open right now, it tells you what files are open
right now, you’re done <3.

It can also tell you what position in the file the process is at, so you
can find out what kind of progress it’s making reading the file.

ngrep / tcpdump

Okay now we’re moving from “super simple tool that does one thing” to
“tcpdump that has a billion options and also this whole BPF berkeley
packet filter business and what is this filter language even”. So I’m
not going to explain how to use tcpdump because I don’t even really
know.

Let’s say you want to

reverse engineer a protocol

find out if there’s really terrible latency or if everything is slow for some other reason

debug why your POST request is formatted wrong in a world before google chrome dev tools

To do all of this, you need to spy on network activity! ngrep and tcpdump
capture packets, let you filter them, and show you what you’re looking for. I’m
not going to explain how to use them here but this ngrep tutorial looks
pretty useful. If you’re looking at output from tcpdump you should probably
dump it to a pcap file and use Wireshark to look at it instead. Wireshark is
the best and way easier to understand because it’s a GUI and it makes
everything pretty for you.

as always with these systems tools, ngrep / tcpdump will tell you The Truth™
about what’s going on on your network.

opensnoop & ftrace

Do you want to know every file your system is opening right now? There’s a
script in Brendan Gregg’s
perf-tools collection that does
that!

I’m mostly including this as an example to show that a lot of stuff is
possible to know – the scripts in that repo don’t work with every Linux kernel
version (I needed to modify it
to get it to work with Linux 3.13). But they use a tracing framework in
the Linux kernel called ‘ftrace’ that can tell you all kinds of stuff.

ftrace seems like quite a bit of work to learn how to use, but also really
powerful. Basically you access it by doing various things to files in
/sys/kernel/debug/tracing, or by using a wrapper command called
trace-cmd. It’s all built into Linux!

atop

atop is like top, but it shows you more stuff and you need to run it as
root. So it’ll show me the CPU & memory usage for each process, but also
how much disk & network I/O it’s doing. It’s neat and a little
terrifying to look at at first (SO MANY NUMBERS).

wow

That’s all for now! If you have other tools you frequently reach for
when trying to figure out what’s going on on your system, I’d be
interested to know what they are. @b0rk on
Twitter, as always :)

an aside – I’ve been thinking about man pages recently, and how you can
read the man page for tcpdump and understand individually all the words,
but it’s not a substitute for someone telling you an Awesome Story about
how they used tcpdump to debug an intermittent certificate problem or a
DNS problem and then they Saved the Day.