useful IT information in small doses

Line-Rate Flow Capture to ELK Stack

Posted on 26 March 2015

Introduction

In certain environments it is necessary to get flow data from different places in your network for compliance or security in general. Recently I ran across a situation in which the native flow gerenators within the Cisco Nexus platform were only able to do 1 in 1000 sampling due to hardware limitations when certain features were enabled. This created a real blind spot, and the requirements were such that a sampling was not good enough. All flows needed to be captured. As always, there were limited budgets, so I had to get creative. I ended up using NTOP's nProbe to generate IPFIX flows to ELK stack via zeroMQ to deliver line-rate flow monitoring solution with no sampling. I should note that this hasn't been pushed to the limit of 10 gig, but so far I have not seen flow drops with considerable load, and according to the nProbe documentation, it should be able to do full 10 gig.

SPAN to nProbe

In order to get the network traffic off the wire, a SPAN port was provisioned on the switch in question and plugged into a mediocre server that had a 10Gig NIC in it.

nProbe Configuration

The nProbe configuration was done via the nBox GUI for the most part and was pretty straight forward. There was one configuration option that I wasn't able to configure with the GUI and I had to just add it to the nProbe configuration file. That was the -i eth0 directive. I'm not sure why I couldn't set this via the nBox GUI, and after mucking around with it for a while, I just directly modified the text config file.