Protocol Information

The virtual MAC is in the format 00-00-5e-00-01-XX where the last octet is filled in by the CARP vhid.

IP Protocol

CARP uses IP protocol number 112 (0x70).

Multicast Advertisements

CARP advertisements are multicast to the 224.0.0.18 or FF02::12 multicast groups when using IPv4 and IPv6, respectively.

TTL/Hop Limit

CARP packets are always sent with a TTL/HLIM of 255 so that CARP packets that have crossed a subnet boundary (i.e., have been passed on by a router) can be recognized and dropped.

Timers

The host that advertises the most frequently will become the master for the CARP group. The timer values configured on each host are sent as part of the CARP advertisements so that all hosts can make an accurate decision as to which host should be master.

Advertisement Interval

This is the base interval at which CARP advertisements will be sent. The default is 1 second and is configured with the advbase keyword. In OpenBSD 5.0 onwards, a value of 0 may be specified if sub-second failover between nodes is required.

Advertisement Skew

This value is used to skew the advertisement interval of a host in order to make it more or less preferred in becoming master. A higher skew value causes a host to send CARP advertisements a fraction of a second slower than hosts with a lower value thereby making it less preferred as the master. The valid range is 0 to 254 with the default being 0. Configure skew with the advskew keyword.

When advbase is set to 0 the skew value alone is used to calculate how often advertisements are sent (the advertisement window) using this formula:

window in microseconds = advskew * 1000000 / 256

Eg: 100 * 1000000 / 256 = 390625 µs

As shown, a skew value of 100 (and interval value of 0) results in an advertisement window of 0.39 seconds.

Failover Timer

If a backup CARP host doesn’t see an advertisement from the master for 3 consecutive advertisement windows then it assumes the master is down. It will take over the CARP group and start advertising itself as the master. The number of advertisement windows to delay before assuming the master is down is hard-coded into CARP and is not tunable.

In the event that two or more hosts have the same timer values configured, the following behavior results:

If preempt is disabled: whichever host starts advertising first (i.e., is configured first) will become the master.

If preempt is enabled: whichever host starts advertising last (i.e., is configured last) will become the master.

The Demotion Counter

Another metric used in determining which host becomes master is the demotion counter. The demotion counter is a value advertised by CARP hosts that announce how “ready” a host is to take on the role of master.

The values used to calculate the demotion counter are stored in dynamic interface groups. By default, each CARP interface is a member of the carp interface group.

When CARP advertises on the network, it takes the sum of the demotion values of all interface groups that the CARP interface is a member of and advertises that as the demotion counter. Hosts with higher values are less preferred to become master for that particular CARP group.

The demotion counter makes it possible to failover selected CARP groups rather than the “all or nothing” approach used with preemption.

Interface States

INIT

All CARP interfaces start in this state. Also, when a CARP interface is admin down it is put into this state. When a CARP interface is admin up, it immediately transitions to BACKUP. Note that in OpenBSD 3.8 and earlier, a bug exists which will cause the host to transition to MASTER right away if preempt is enabled.

BACKUP

The host is listening for advertisements from the master. If no advertisements are seen after 3 advertisement windows then assume the master is down, transition to MASTER state and start sending advertisements. If an advertisement is seen with a worse (higher) advertisement window than ours, and if preempt is enabled, transition to MASTER and start sending advertisements.

MASTER

The host is forwarding traffic directed to the virtual/group IP address. The host is also sending advertisements once per advertisement window that announce its presence to other CARP hosts within the CARP group. The host still listens for advertisements from other CARP hosts. If an advertisement is seen with a better (lower or equal to ours) advertisement window, transition to BACKUP and allow the other host to become MASTER.

Note that changing any values associated with a CARP interface (timers, password, etc) will automatically result in the interface being put into the INIT state.

Under normal circumstances, there can be multiple hosts within a CARP group in the BACKUP state, but only one host will ever be in MASTER state.

CARP Bugs

Preempt Failover Race

In the following scenario, a race occurs: Two firewalls each connected to switches using two separate physical interfaces on the firewall. The firewalls have preempt enabled; fw01 is master, fw02 is backup on all CARP groups. One of the switches goes away causing the interfaces on the firewalls to go down. Since both firewalls have preempt enabled, they up their advskew to 240 on all remaining CARP groups. Here’s the race: Who becomes master now? There was an interesting discussion about this on the OpenBSD pf mailing list here: http://marc.theaimsgroup.com/?l=openbsd-pf&m=113881646826219&w=2. Steven S commented that his workaround is to only set preempt on the master firewall and to disable it on the other firewall(s). There is no permanent solution to this issue currently.

4 thoughts on “OpenBSD CARP Notes”

Awesome post! I just started messing with pfsense and my first project is setting up failover. I knew nothing except the historical context of CARP, but now I’ve got a page of notes that’ll surely help going forward.

That’s a smart question. As far as I can see packets are always sent to the virtual MAC (even when running in load balance mode). When a failover occurs, the CARP node that takes over for the group will send a gARP for the vIP and vMAC to update L2 forwarding tables across the network. Note that I do need to update this post (probably in more ways than just this) to reflect that in balance mode, CARP will use a multicast MAC as the vMAC which forces the network to flood traffic destined to the vMAC out all switch ports.