Redesign of the pfsync Protocol, Part 1

David Gwynne (dlg@) recently committed a rewrite of OpenBSD's pfsync protocol. This project fulfilled part of his studies in the School of Information Technology and Electrical Engineering at the University of Queensland. I was fortunate enough to get a chance to read David's paper which detailed his research into, and eventual redesign of, the pfsync protocol. These changes result in a much more efficient synchronization mechanism between OpenBSD cluster nodes.

It is our great honor to be able to bring you David's paper as a series of articles for the OpenBSD Journal. This week we start off with a brief introduction to his work and some of the background behind pfsync. Enjoy!

Active-Active Firewall Cluster Support in OpenBSD

Abstract

The OpenBSD UNIX-like operating system has developed several technologies that make it useful in the role of an IP router and packet filtering firewall. These technologies include support for several standard routing protocols such as BGP and OSPF, a high performance stateful IP packet filter called pf, shared IP address and fail-over support with CARP (Common Address Redundancy Protocol), and a protocol called pfsync for synchronisation of the firewalls state with firewalls over a network link. These technologies together allow the deployment of two or more computers to provide redundant and highly available routers on a network. However, when performing stateful filtering of the TCP protocol with pf, the routers must be configured in an active-passive configuration due to the current semantics of pfsync. I.e., one host filters and routes all the traffic until it fails, at which point the backup system takes over the active role. It is possible to configure these computers in an active-active configuration, but if a TCP session sends traffic over one of the firewalls and receives the other half of the connection via the other firewall, the TCP session simply stalls due to a combination of pf's stateful filtering and pfsync being too slow to cope. This report documents an attempt at solving this problem by modifying the pfsync implementation and protocol to improve support of filtering OpenBSD routers in active-active configurations.

Introduction

OpenBSD is becoming increasingly popular as a platform for network security services, in particular network routing and firewalling. Currently there is support for building redundant firewalls with OpenBSD using pf and pfsync in active-passive configurations, however, to scale throughput between networks protected by OpenBSD firewalls it should be possible to configure them in active-active configurations. This project is an attempt at supporting active-active OpenBSD firewalls using pf and pfsync.

This document tries to give some context to the problems with building active-active firewalls with OpenBSD currently, then goes on to explain the changes that were made to enable it.

It assumes some familiarity with the C programming language, concepts relating to the
workings of operating system kernels, and the IP network protocols including TCP.

Background

OpenBSD

OpenBSD is a general purpose UNIX-like operating system. As the name implies, it is
descended from the long heritage of the Berkeley Software Distribution (BSD for short),
which began as a set of modifications to the original AT&T UNIX and eventually grew into
a completely independent code base.

The OpenBSD Project was created in late 1995 by Theo de Raadt who wanted to create a
free and open source distribution with a focus on "portability, standardisation, correctness, proactive security, and integrated cryptography". While it is usable for a variety of tasks ranging from a desktop system up to file and print services, it is this focus on correctness and security that has let OpenBSD evolve into a highly usable and secure operating system for network security services. OpenBSD ships with a variety of software and capabilities in the base system that allow administrators to build very capable routers and firewalls.

OpenBSD As A Router

OpenBSD can be installed on a wide variety of hardware and with some simple configuration changes can be used as a standard router on IPv4 and IPv6 networks. It
also includes support for technologies such as IPsec, Ethernet VLAN (802.1Q) tagging, a
wide variety of network tunneling protocols, a high quality firewall, IP and Ethernet load balancing, and can interoperate with other routers using protocols such as OSPF and
BGP.

pf - The OpenBSD Packet Filter

For the majority of its existence, OpenBSD has shipped with some form of packet filtering software that can be used to enforce network security policy at the IP and TCP/UDP layers for traffic flowing through an OpenBSD box. Originally the codebase included in OpenBSD was Darren Reeds' IPFilter. However, in 2001 an issue with the license on that code caused it to be removed from the OpenBSD source tree.

Fortunately, shortly after its removal a developer named Daniel Hartmeier began
implementing a new packet filter simply called pf. The code was quickly imported into the OpenBSD system, and development continued rapidly. Since that starting point pf has
developed into an extremely capable and fast firewall.

The network policy that pf is to enforce is described in a ruleset which is parsed and
loaded into pf inside OpenBSD. These rulesets are generally just a list of pass or block rules that let you match against packets based on their address family, protocol, and if
available their port number. However, pf is also able to perform actions such as
normalising fragmented IP packets, and network address translation (rewriting IP
addresses or port numbers inside a packet). Once loaded into the kernel the ruleset is
then evaluated for every packet that enters or leaves the system. Packets going through
an OpenBSD based router will be evaluated by pf twice, once coming into the network stack and again when it is leaving the stack.

Iterating over the ruleset for each and every packet going into the system can be slow
though, but fortunately pf is a stateful firewall. This means that pf will try to keep track of connections going through itself. States use characteristics of the packet such as the address family, IP addresses, port numbers, and in the case of TCP, the state and
sequence numbers of the connection. These states are organised into a red-black tree
inside the kernel, and packets entering pf are compared against existing states before the ruleset is evaluated. If a packet matches a state, pf doesn't evaluate the ruleset and simply allows the packet though.

This means there are three benefits to the pf state table.

Firstly, there is an O(log n) cost for state lookups for a packet (where n is the number of states) vs an O(n) cost for a ruleset evaluation (where n is the number of pf rules). Since most packets will belong to an existing state they will therefore tend to match existing states, providing a huge speedup in the per packet evaluation costs.

Secondly, because pf does have some state about a connection it is able to independently check that a packet does in fact belong to an existing connection, and is not some spoofed packet with similar characteristics (ie, IP or port numbers).

Thirdly, because the state table will allow packets through for existing connections, the ruleset only has to concern itself with matching packets that create connections. This is another security benefit since without the state rules would have to be written that allow packets in both directions through a firewall. To highlight the difference, here is an example ruleset for TCP going through pf that does not take advantage of the state table:

pass in inet proto tcp from any port >1024 to $webserver port 80 no state
pass out inet proto tcp from $webserver port 80 to any port >1024 no state

As you can see this ruleset has to allow TCP replies from the web server back through the firewall otherwise the connection will fail. A ruleset taking advantage of the state table would look like this:

pass in inet proto tcp from any to $webserver port 80 flags S/SA

This rule only allows TCP packets with the SYN flag set (out of the SYN/ACK pair) to the webserver through the firewall. Pf by default will create a state for this connection and will automatically allow replies just for that connection back through.

Pf is quite scalable, meaning that with a suitable machine you can easily run firewalls with many network interfaces and with thousands of firewall rules. The number of states the firewall will manage is dependent largely on the amount of memory in the system. It is common for pf to be managing tens of thousands of concurrent state entries.

pfsync

While it has been established that stateful firewalling is good and a necessary thing for pf to operate at speed, it is unfortunately not a good thing in the event of a firewall failing. For simply routing packets it is trivial to configure two routers with the same configuration and in the event of one failing you can simply replace the failed unit with the functional one.

There are several systems in protocols to automate that failover, indeed, OpenBSD
features several such as CARP (Common Address Redundancy Protocol), ospfd (a routing
daemon implementing the OSPF protocol), and bgpd (a BGP implementation). These systems allow a router to take over from a failed peer within a matter of seconds.

However, if these routers are running pf then existing network connections through the
redundant peer will no longer work since it has no knowledge of the state table the failed firewall had built. To compensate for that the pfsync protocol was developed.

Pfsync simply sends messages between pf firewalls that notify each other of state
inserts and deletions, and most importantly, state updates. Because pf keeps track of the TCP sequence numbers it is important for it to notify its peers when those sequence
numbers progress so in the event of a failure the peer knows where the TCP connection was up to.

Pf with pfsync allows the deployment of fully redundant stateful firewalls. In the event of failure the redundant firewall will have a full knowledge of the current state table and will be able to continue forwarding packets for those existing connections. However, because of a semantic in the implementation of the pfsync protocol, it is currently not possible to build clusters where both firewalls are actively able to handle traffic.

Active-Active Routers and Firewalls

It is becoming more common to deploy multiple routers between networks to increase the
forwarding performance between those networks. Two routers are obviously twice as capable as a single router at forwarding traffic.

Since OpenBSD can be deployed as a firewall and router, it would be nice to be able to
increase performance of such a deployment by simply adding firewalls as necessary.
However, as stated above, this is currently not possible. Why this is the case is described in detail below.

[ To be continued... ]

(Comments are closed)

By
Anonymous Coward ()
on 2009-02-20 02:34

this is great. i want to read the next part!
to pf-book-writers: make this text a preamble

By
Anonymous Coward ()
on 2009-02-20 02:51

If I were to submit a few articles, how do you do those [code] blocks - the gray and white text boxes. What's the HTML code I would need to use? =)

> If I were to submit a few articles, how do you do those [code] blocks - the gray and white text boxes. What's the HTML code I would need to use? =)
>
> Looking forward to Part II.

Just view the source. :) They're blockquotes, but I don't think you can submit them normally through the "Add Story" page. One workaround is to format everything like you want and then wrap it in pre tags. Make sure to let us know that's your intent.