Richard Bejtlich's blog on digital security, strategic thought, and military history.

Friday, January 28, 2005

Flowgrep: Flow-oriented Content Matching

Last month I found Meling Mudin's IDS blog, and learned of Jose Nazario's tool Flowgrep. Flowgrep is written in Python. It is similar to Ngrep, which I addressed in my first book. Ngrep is packet-oriented, meaning the strings for which Ngrep searches must all appear in a single packet. If you search for 'bejtlich', and 'bejt' is in one packet and 'lich' another, then Ngrep won't find anything.

Flowgrep, in contrast, is conversation-oriented. Flowgrep assembles TCP sessions, as well as pseudo-sessions for UDP and ICMP. Flowgrep will rebuild a conversation where 'bejt' is in one packet and 'lich' another, and report seeing 'bejtlich'.

Flowgrep relies on Mike Schiffman's Libnet and Mike Pomraning's Pynids, a Python wrapper for Rafal Wojtczuk's Libnids. Mike was kind enough to work with me over the last week to get Pynids operational on FreeBSD 5.3.

Here's how I ended up with a working Flowgrep implementation. First I ensured Python was on my system:

Now that Flowgrep is installed, let's see how it works. For demonstration purposes, I have telnet running on host janney on an odd port -- 47557 TCP. I tell Tcpdump to collect all traffic on janney involving that port. This gives us traffic for later analysis:

"So what," you might say. I can reconstruct streams with Tcpflow. True enough, but how did we end up with these two streams? These are the result of Flowgrep searching streams for the content 'hackerpassword'. How did the packets which built that stream look? Let's jump to the packets where the server presented "Password:" to the client:

By now everyone should appreciate just how powerful and useful Flowgrep can be. Even though every character of my 'hackerpassword' string appeared in separate packets, Flowgrep assembled the stream and logged the traffic that matched the filter I specified.

Flowgrep is not the only tool with this capability, since more robust intrusion detection systems offer similar features. However, this is the only stand-alone tool I know that offers rapid string matching on stream contents on arbitrary ports.

I chose to demonstrate telnet because I knew it would place virtually every character I typed into separate packets. The principle applies anywhere you are concerned that content of interest could be split between multiple packets. Furthermore, Flowgrep is designed to watch UDP and ICMP conversations as well.