OpenVPN/iproute2/iptables — Part 2

May 7, 2011

I had blogged about this some time ago. The configuration I described in that post worked fine on my laptop, with Debian installed, but when I tried it on my Desktop, where I use Gentoo, it wouldn’t work.

It took me *3 days* of ‘debugging’, until I was able to find why that happened!

I tried various changes to the iptables and iproute2 configuration, giving more hints to both utilities in order to use the correct routing table, mark the packets correctly etc, but it still wouldn’t work.

After a lot of time tweaking the configuration, without results, I saw that, although ping -Ieth0 ${VPN_SERVER}, didn’t ‘work’ (with openvpn running, and tap0 configured with the correct address/netmask), I could see with tcpdump the ‘ECHO REPLY’ packets sent by the VPN server, with correct source and destination addresses.

After stracing the ping command, I saw that when ping issued a recvmsg syscall, recvmsg returned with -EAGAIN. So, now I know that the packets do arrive to the interface with correct addresses, but they couldn’t ‘reach’ the upper network stacks of the kernel.

The problem was that both machines were running vanilla kernels, so I couldn’t blame any Debian or Gentoo specific patches. But since I knew that the problem was in the kernel, I tried to see if any kernel .config options, regarding NETFILTER, and multiple routing tables didn’t match between the two configs. But I couldn’t find anything that could cause that ‘bug’.

So since the kernel sources are the same, and I can’t find anything in the .configs that could cause the problem, I try tweaking some /proc/sys/net ‘files’, although I couldn’t see why these would differ between the two machines. And then I saw some /proc/sys/net/ipv4/ files in Gentoo, that didn’t show up in Debian (/proc/sys/net/ipv4/cipso*).

I googled to find what cipso is, and I finally found out that it was part of the NetLabel project. CIPSO (Common IP Security Option) is an IETF draft (it’s quite old actually) and is implemented like a ‘security module’ in the Linux Kernel, and it was what it caused the problem, probably because it tried to do some verification on the inbound packets, which failed, and therefore the packets were ‘silently’ dropped. LWN has an article with more infromation about packet labeling and CIPSO, and there’s also related Documentation in the Linux Kernel.

make defconfig enbales Netlabel, but Debian’s default configuration had it disabled, and that’s why Openvpn/iproute2/iptables configuration worked with Debian, but failed on Gentoo.

Instead of compiling a new kernel, one can just do
echo 0 > /proc/sys/net/ipv4/cipso_rbm_strict_valid

and disable CIPSO verification on inbound packets, so that multiple routing tables and packet marking work as expected.