Author
Topic: pfSense 2.0.1 locking up (hard) (Read 3391 times)

I've just installed a new pfSense setup for my client using one of the SuperMicro Atom D525 motherboards.

It has 3 NIC's now. 2 on motherboard and one Intel NIC plugged in via PCIe.

It's been running flawlessly at home where I built a similar (but not completely same) model of his network.. but when I got it to the client... and we decided to add RIP to the system as it now sits between his netscreen firewall and his ExtremeNetworks core router, the unit would randomly lock up. It requires a hard reset. (local keyboard is completely unresponsive)

(or so it seems. I've turned off RIP for now since the network is pretty static)

And it's now the waiting game.

ALL this system is doing is Squid proxy. There's really no other rules for the FW/NAT except "manual NAT" and "allow all" for the firewall. The setup is pretty simple/no-frills.

I have a SM-based pfsense with a D510 and a pair of Intel (em) on-board NICs. MBUF usage grows continually until exhausted, followed by a full lockup. The "kern.ipc.nmbclusters="131072"" setting puts your MBUF limit much higher than default and will postpone or prevent lockup, depending on how much you need. I think you can always set that number higher if you have the RAM to back it.

I have a SM-based pfsense with a D510 and a pair of Intel (em) on-board NICs. MBUF usage grows continually until exhausted, followed by a full lockup. The "kern.ipc.nmbclusters="131072"" setting puts your MBUF limit much higher than default and will postpone or prevent lockup, depending on how much you need. I think you can always set that number higher if you have the RAM to back it.

Thanks for the note... it's good to hear from people with similar hardware.

I have the same MB at home (not for this install though, I got the D525 for the client) -- and THEN bought the D510 since I had parts for it laying around and don't need the mem speed for it.

Your proposition gives me the willies. What? Am I supposed to schedule a reboot every weekend? Turn on the BMC watchdog? (need to test that at home first) Yikes.

I do have 4GB in the unit -- but I am running pfSense-2.0.1-embedded_AM64 off an SSD... so RAM isn't exactly wild-free-and-open. But, I'll bump up the bufs later.

right not seems to have a fairly static 5574/25600 and the client is getting their full 50Mb/s BW to the internet. So overall, provided the system stays up, we're pretty happy.

Theoretically the MBUF usage should level off at some point. Mine are at 34652/131072 and growing with 21 days of uptime. This is approximately the same uptime I had 21 days ago when one of the NICs stopped passing traffic, so I have yet to see my MBUF usage level off on this system. Mind you I have hundreds of users, so I can only guess that that is a factor, because my internet connection is less than 40 Mbps.

I should add that increasing your max MBUFS is not going to put much of a dent in 4G of RAM. My system has 4G and no swap and is currently reporting 10% memory usage. My current states are reporting at 11795/389000 but I've seen them as high as 34000. I think memory is allocated on your max rather than current, so you can see that increasing your nmbclusters isn't likely to be a problem on your system.

Theoretically the MBUF usage should level off at some point. Mine are at 34652/131072 and growing with 21 days of uptime. This is approximately the same uptime I had 21 days ago when one of the NICs stopped passing traffic, so I have yet to see my MBUF usage level off on this system. Mind you I have hundreds of users, so I can only guess that that is a factor, because my internet connection is less than 40 Mbps.

Hmmm...

This user has 50Mb/s and about 3,000 PC's (Mac/Windows Mix - but mostly probably 90% Mac).

I just need to watch it. So far, the CPU has been pretty sleepy according to RRD. Peaks of 25% with most daily usage under 20%.(I'm assuming that's system total consider this is a dual-core w/HyperThreading mb)

Very nice.

The state table I've seen peak around 20,000 of 390,000. So.Also nice.

Thanks for the data. I'll keep sharing as I watch this systems first day in "production" mode. (scaaaarrryyy... hahahah)

I should add that increasing your max MBUFS is not going to put much of a dent in 4G of RAM. My system has 4G and no swap and is currently reporting 10% memory usage. My current states are reporting at 11795/389000 but I've seen them as high as 34000. I think memory is allocated on your max rather than current, so you can see that increasing your nmbclusters isn't likely to be a problem on your system.

Agreed. I didn't think adding more MBUFs would be a big impact either. (thanks for the note though)

I have a SM-based pfsense with a D510 and a pair of Intel (em) on-board NICs. MBUF usage grows continually until exhausted, followed by a full lockup. The "kern.ipc.nmbclusters="131072"" setting puts your MBUF limit much higher than default and will postpone or prevent lockup, depending on how much you need. I think you can always set that number higher if you have the RAM to back it.

Forgive my possibly noobish question, but how do I set this setting?

Do I just add it on the: "System -> Advanced -> System Tunables" page?

I have a SM-based pfsense with a D510 and a pair of Intel (em) on-board NICs. MBUF usage grows continually until exhausted, followed by a full lockup. The "kern.ipc.nmbclusters="131072"" setting puts your MBUF limit much higher than default and will postpone or prevent lockup, depending on how much you need. I think you can always set that number higher if you have the RAM to back it.

Forgive my possibly noobish question, but how do I set this setting?

Do I just add it on the: "System -> Advanced -> System Tunables" page?

You have to look at the link above. There's a file in /boot you need to create and put those parms in.

Quote

Does this setting require a reboot of the system?If no, does changing the setting drop any current states?

Yes. it needs a reboot. (so that should answer the 2nd part of that question)

It worked for me with no issues.. and I'm seeing a level off at about 7200 (7174) of 131072 after a day over heavy operation.

Some settings can be set from the system tunables page, however this one cannot. You must add it to /boot/loader.conf.local (create the file if it does not exist) and then reboot for it to take effect. Do not add it to /boot/loader.conf, as this will be overwritten on the next system upgrade.