Basically I'm not a kernel developer. I can read the kernel panic and see what syscall is causing shit to go down, and can plainly see how its' happening across two entirely different kerenl builds. My expectation is that this is an actual bug either in the netfilter code itself, or with something grsecurity is doing in conjunction with it.

For background, this runs as a tor exit node which quite happily pushes 20-50 thousand packets per second. The firewall rules are reasonably simple, and I am only invoking the conntrack module twice:

Unfortunately something on the order of one in a billion packets is murdering this server. And its' happening every 12 hours or so, which is "annoying".

This panic is happening on 3.17.1, but also on 3.16.5 as well but for brevity's sake I'm not pasting that as well. I'd do a bug report to Gentoo hardened but this feels a bit out of their depth at the moment.

Note:

* Ignore the xt_* modules. They are not in use and the panics predate them. Besides, I haven't yet made the CHAOS target behave the way I want yet.* netconsole produces staggered output, but at least it works. Yes, you can trap kernel panics remotely with netconsole!* kernel config cliffs notes: grsec automatic config, usage server, no virtualization, security priority, selinux is in use but permissive due to "tuning issues".

Now, I'm more than happy to help participate in squishing this and will provide whatever is needed. I just need a solid push in the right direction.

this looks like another use-after-free bug (rbx was dereferenced which holds the pattern used by SANITIZE) but i'm afraid we won't be able to figure it out either. someone with deep knowledge of how rcu locking is supposed to work in conntrack would have to look at it (the triggering function seems to have the proper locking so it's some other user of conntrack data that is probably missing some rcu locking). you could also run "addr2line -e vmlinux -fip ffffffff814b58ce" on your kernel to have a better location info for the actual triggering code line (best is if you build your kernel with CONFIG_DEBUG_INFO and CONFIG_DEBUG_INFO_REDUCED).

The option CONFIG_DEBUG_INFO is already set, and it appears that REDUCED only reduces (imagine that) the information output. I wouldn't have any output otherwise, if I'm understanding the debugging properly.

Would any of the RCU debugging options (or anything else for that matter) in the kernel help isolate this thing further?

I don't mind going on a bughunt. Its' just that my kernel debugging skills are very limited.

Where next with this?

Mostly I am trying to suss out whether this is a grsecurity bug, gentoo bug (a small amount of the Gentoo hardened patchset is Gentoo-specific and not just grsecurity), or is an issue with the base kernel itself. Or some bizarre interaction between a combination thereof.

If this is a netfilter issue, or at least convincingly not a grsecurity issue, what would be my best way to proceed?

what you are seeing is the SANITIZE feature catching a use-after-free bug which must exist in the vanilla kernel already as we don't touch conntrack code beyond some trivial constification related changes. if you ported SANITIZE to vanilla or enabled slab debugging with proper magic values (the free poison value must have its LSB set to 0 to trigger this particular bug which we were (un)lucky to choose for our poison value) then you'd trigger it on vanilla i'm sure. if you want to keep SANITIZE but sweep this particular bug under the carpet then you can add SLAB_NO_SANITIZE to the kmem_cache_create call in net/netfilter/nf_conntrack_core.c:nf_conntrack_init_net.

If so, there are good chances to report this upstream as a use-after-free bug, referring to the patch, that, essentially, just does a memset(0xfe) on a kmem_cache_free(ptr). After that function has been called, no code should access *ptr any more, but, apparently, some code does and triggers the #GP you're seeing.

Ah, ok, thanks for the explanation. The SANITIZE sidebar now makes sense now.

The feature isn't critically important, but is nice to have. Other stuff is far more important. But squishing this would be best especially since it has revealed an issue in the vanilla kernel. Kernel bugs in networking code make me nervous.

Just for clarity, this is the build plan for that patch for reproducability's sake:

minipli's patch's against vanilla, not grsec and it unconditionally enables the extracted portion of SANITIZE, you don't need to configure anything special for it (a make oldconfig on the grsec config should produce a good config for the vanilla kernel).

What I find interesting is that you don't always get netconsole output. This panic, for example, did not output via netconsole. But I did get other trash messages before the panic, indicating the logging method worked.

I wonder why.

Anyway, issue confirmed. What's next?

I'm going back to the grsec kernel with sanitize disabled in the short term, but getting this squished would be optimal.

a full log with register content would still be nice but it's the same instruction that triggers so i'm fairly sure it's the same bug. as for what's next, it's clearly an upstream problem so you have the honour to report it on lkml/netdev and get it fixed . probably confer with minipli as well since it's his code in SANITIZE that found this one and the other one somewhere in UDP.

I know you'd like better output, but how ARE you supposed to get full output in situations like this?

I've battled this for years! I've gone through the sysrq functions and there isn't anything I can use to "scroll up" in the buffer. The only option seems to be some value of directed console output (serial, network) which as you can see is a bit finiky in the latter case. I mean, I can enable a framebuffer device that's "bigger" so I can screenshot output but that's a halfassed solution at best. I've worked in linux admin for years and I'm yet to see a good solution to this, and netconsole is something I've never seen anyone do for panic tracking.

I'll work on a kernel bug report later.

Though I'm sitting here wondering why I'm the one who found this because my kernel use cases are almost always bog standard and well traveled. Probably the unique combination of grsec, the memory saniziation option (I specifically enabled it), huge packet throughput, and netfilter connection tracking.

As pipacs said, next would be to report this to netdev@vger.kernel.org. You should mention the above patch, that sanitizes freed slab objects with the pattern '\xfe' which can be seen in registers when the panic happens -- assuming you can trigger the panic again with a full backtrace and register dump. But as it happend in the past, it should be very likely.