Wednesday, October 19, 2011

When is full packet capture NOT full packet capture?

I was looking at some packets recently and noticed the Wireshark message "Packet size limited during capture". This was strange since the packets came from a Sguil sensor performing full packet capture using Snort's default snaplen on a standard Ethernet connection (no Jumbo frames and no VLAN tags). Drilling down into the packet capture, some of the packets were 2900 bytes and Snort was only capturing the first 1500 bytes. The full packet capture was not "full" packet capture after all.

I won't repeat all the information in those links, but I'll summarize by saying that the NIC was reassembling packets before being passed up the stack to Snort. I disabled the offload features and then verified that this resulted in no more packets larger than 1500 bytes. The packet capture truly was "full" packet capture.

[ UPDATE: A reader asked why we couldn't simply change Snort's default snaplen to a larger value to capture the 2900-byte packets. While it's true that would solve the "full" packet capture problem, another problem would remain. Since the packets are being reassembled on the NIC, Snort is not able to properly perform target-based reassembly (see the Snort manual link above). This opens the door for potential IDS evasion/insertion attacks. NIC offload settings need to be disabled so that Snort sees the same packets the destination host does. ]

Some NIC/driver combinations may disable these offload settings by default, while others enable it by default. You should check your sensors now before you get into a situation where you really need that full packet capture and find out that you don't actually have it. To check, run ethtool with the "-k" (lower-case k) option on the interface you'd like to check. For example, to check eth0:

These settings will remain in effect only while the OS is booted, so this needs to be applied at every boot. This can be done by adding the above for-loop as a "post-up" command for each of the interfaces in /etc/network/interfaces. If you're still using the graphical Network Manager to configure your interfaces, I've put together some documentation on disabling it and configuring interfaces and their offloading features via /etc/network/interfaces:

Did you notice any difference in performance after disabling the offload features?

Is there a better way of disabling offload features globally? I tried putting the commands in /etc/rc.local and /etc/init/securityonion.conf, but the only way I could get it to work consistently was via /etc/network/interfaces as documented above.

I'm considering disabling offload features by default in the Security Onion Setup script. Can anyone think of any reason why this might be a bad idea?

My Snort machine that I'm going to install SO on, has the following defaults, Intel gigabit card:

Features for em1:rx-checksumming: ontx-checksumming: on tx-checksum-ipv4: off [fixed] tx-checksum-ip-generic: on tx-checksum-ipv6: off [fixed] tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: off [fixed]scatter-gather: on tx-scatter-gather: on tx-scatter-gather-fraglist: off [fixed]tcp-segmentation-offload: on tx-tcp-segmentation: on tx-tcp-ecn-segmentation: off [fixed] tx-tcp6-segmentation: onudp-fragmentation-offload: off [fixed]generic-segmentation-offload: ongeneric-receive-offload: onlarge-receive-offload: off [fixed]rx-vlan-offload: ontx-vlan-offload: onntuple-filters: off [fixed]receive-hashing: onhighdma: on [fixed]rx-vlan-filter: off [fixed]vlan-challenged: off [fixed]tx-lockless: off [fixed]netns-local: off [fixed]tx-gso-robust: off [fixed]tx-fcoe-segmentation: off [fixed]fcoe-mtu: off [fixed]tx-nocache-copy: onloopback: off [fixed]rx-fcs: offrx-all: off

This is good stuff. On CentOS/RHEL you can create a /sbin/ifup-local file, then add those commands to it. The /etc/sysconfig/network-scripts/ifup-post script looks for a /sbin/ifup-local and if there runs the commands on boot within that file. I have other offloading showing (rxhash, rxvlan, and txvlan) but I am not sure if others do as well. This was very helpful. thanks. Here is what I added to my /sbin/ifup-local

Security Onion

Security Onion is a Linux distro for intrusion detection, network security monitoring, and log management. It's based on Ubuntu and contains Snort, Suricata, Bro, OSSEC, Sguil, Squert, ELSA, Xplico, NetworkMiner, and many other security tools. The easy-to-use Setup wizard allows you to build an army of distributed sensors for your enterprise in minutes!