We have an OpenBSD router at each of our locations, currently running on generic "homebrew" PC hardware in a 4U server case. Due to reliability concerns and space considerations we're looking at upgrading them to some proper server-grade hardware with support etc.

These boxes serve as the routers, gateways, and firewalls at each site. At this point we're quite familiar with OpenBSD and Pf, so hesitant at moving away from the system to something else such as dedicated Cisco hardware.

I'm currently thinking of moving the systems to some HP DL-series 1U machines (model yet to be determined). I'm curious to hear if other people use a setup like this in their business, or have migrated to or away from one.

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
If this question can be reworded to fit the rules in the help center, please edit the question.

1

I found the answers helped us as we have been running open bsd for 9 years and began to think to move to jos because of power issues in the data centre. Now I will think again as I think we have undervalued the benefits of running on an open platform.
–
user105680Jan 2 '12 at 11:09

11 Answers
11

We run exclusively OpenBSD routers/firewalls to serve FogBugz On Demand. Unless you're operating in a transit role and need the extremely high pps throughput that purpose-built hardware and integrated software can provide, OpenBSD on solid hardware will be a more manageable, scalable, and economical solution.

Comparing OpenBSD to IOS or JUNOS (in my experience):

Advantages

The pf firewall is unmatched in terms of flexibility, manageable configuration, and integration into other services (works seamlessly with spamd, ftp-proxy, etc). The configuration examples do not do it justice.

You get all the tools of a *nix on your gateway: syslog, grep, netcat, tcpdump, systat, top, cron, etc.

You can add tools as necessary: iperf and iftop I've found very useful

So long as you're not talking about backbone routers in an ISP-like environment or edge routers interfacing with specialized network connections, OpenBSD should be just fine.

Hardware

The most important thing to your router performance is your NICs. A fast CPU will quickly get overwhelmed under moderate load if you have shitty NICs that interrupt for every single packet they receive. Look for gigabit NICs that support interrupt mitigation/coalescing at least. I've had good luck with Broadcom (bge, bnx) and Intel (em) drivers.

CPU speed is more important than in dedicated hardware, but not something to fret about. Any modern server-class CPU will handle a ton of traffic before showing any strain.

Grab yourself a decent CPU (multiple cores don't help much just yet, so look at raw GHz) good ECC RAM, a reliable hard drive, and a solid chassis. Then double everything and run two nodes as an active/passive CARP cluster. Since 4.5's pfsync upgrade you can run active/active, but I haven't tested this.

They've been rock-solid since deployment. Everything about this is overkill for our traffic load, but I've tested throughput upwards of 800Mbps (NIC-limited, the CPU was mostly idle). We make heavy use of VLANs, so these routers have to handle a lot of internal traffic too.

Power efficiency is fantastic since each 1U chassis has a single 700W PSU powering two nodes. We've distributed the routers and balancers through multiple chassis so we can lose an entire chassis and have pretty much seamless failover (thank you pfsync and CARP).

Operating Systems

Some others have mentioned using Linux or FreeBSD instead of OpenBSD. Most of my servers are FreeBSD, but I prefer OpenBSD routers for a few reasons:

A tighter focus on security and stability than Linux and FreeBSD

The best documentation of any Open Source OS

Their innovation is centered around this type of implementation (see pfsync, ftp-proxy, carp, vlan management, ipsec, sasync, ifstated, pflogd, etc - all of which are included in base)

FreeBSD is multiple releases behind on their port of pf

pf is more elegant and manageable than iptables, ipchains, ipfw, or ipf

Leaner setup/install process

That said, if you're intimately familiar with Linux or FreeBSD and don't have the time to invest, it's probably a better idea to go with one of them.

Thanks for the extremely detailed reply. What you describe is pretty much exactly the type of system we're looking at building, a pair of servers with onboard dual GigE and and a dual GigE add-in NIC in a CARP failover configuration. It's very reassuring to see that someone else is running such a setup in a major production system.
–
Kamil KisielJun 29 '09 at 22:28

1

Personally I prefer iptables, I think pf is too restricted. My experience with CARP on OpenBSD is that it's great when you want to do planned jobs (planned failover), but the failover will most often not work when there is an actual fault. I've had exactly one successful pf crash failover, and this was with OpenBSD 4.5. Also, the support situation for OpenBSD is dismal. If you don't have the knowledge in-house or pay someone then the answer to all questions or support when it crashes is: "you mother is fat".
–
ThomasJun 30 '09 at 6:32

1

I run pf/pfsync/CARP two firewalls in a failover config. I've experienced two failover situations and in both cases I only learned about it from my monitoring system telling me one of the firewalls was down. The cluster's services continued without noticeable interruption.
–
InsyteAug 12 '09 at 15:08

pfsense Is a great FreeBSD based firewall, its very feature rich, easy to setup, and has an active community as well as support options. There are several people using it in commercial / production situations that are active in the forum. I use it at home and I'm pushing it at work, its a really well put together alternative. They even have a VM image for download to test it out with!

i looked at that link. that variant of MonoWall looks great. :-)
–
djangofanAug 12 '09 at 15:22

I believe mono focuses on embedded hardware, whereas pfsense focuses on pc based systems. I believe it was intended to offer more advanced/enterprise-class features than those found in m0n0wall or other basic firewall distro's.
–
ChanceSep 24 '10 at 15:02

For my father's small business with one branch office, I use OpenBSD as the router/gateway/firewall for both the main and branch office. It has never let us down. We use a Dell Tower Server at each location. Each server is equipped with a Dual GiGE card, 8GB of ram (slight overkills, I know) and works well. The branch office is configured to connect to the main one through IPSEC and OpenBSD's IPSEC implementation is delightfully easy to use.

I have been running OpenBSD (4.9) in production on our main firewall for quite some time. Its a rather old ASUS MB with 2 GB DDR (1) RAM and a dual core (2 GHz) Athlon. I bought a quad port intel card (pci-express) and used in the x16 graphics port. Do NOT throw away your PCI graphics cards if you have any laying around. You will need it as a graphics card if you plan on using the 16x PCI-express port for the NIC (onboard gfx didnt work in my case).

I know its not "Enterprise class" hardware. but these are the clear benefits of this setup:

I have a lot of these MB lying around, and thus will never run out of spare parts (getting ready for CARP also).

Most cheap AMD bords support ECC RAM!.

All hardware/spare parts are "of the shelf" cheap and stable

Performance on these rigs are great (4x Gbps), even for our rather heavy hosting setup!

OpenBSD gateways are used in many enterprise setups. We have two OpenBSD gateways on our networks.

I still recall one funny episode with OpenBSD: the hard disk died, but the gateway just carried on routing traffic, like nothing had happened, serving from memory alone. It gave me some time to setup another instance.

Very low hardware requirements, Dual Opteron 248's are great. I rarely see the cpu go over 5%. They are very stable. I've being using it just over 7 years now with no issues.

I have in the past. I installed it originally on some "whitebox" PC's, then upgraded to a Dell Power Edge 2950. Redundant power supplies, hard drives - big improvement from a reliability standpoint. Not an observed improvement of course, we got lucky and the whitebox never crashed, but theoretically we were in better shape with more redundancy.

We were only using it to packet filter a T1, so not a noticeable performance improvement.

Did you considered switching to FreeBSD?
OpenBSD can't fully utilize modern SMP systems (i.e Core2Quad).
FreeBSD has pf and ipfw that you can use simultaneously and also has non-GIANT networking layer.

We've been running software FreeBSD routers as ISP gateways for years, this saved us a lot of $$

I can't speak for *BSD (yet...give me time...) but we've been running Linux routers for 10+ years and love them. Cheaper, no license hassles, and if you look at the docs you'll find you have most of the tools you need to get things done. I would suspect that BSD is very much in the same boat.

We're running a DL365 G1 with a single processor socket filled and 6Gb, although the RAM is mostly for servicing mailboxes...