4. The CARP protocol

CARP (Common Address Redundancy Protocol) is the protocol that achieves system redundancy, by sharing an IP address across a group of hosts on the same network segment (redundancy group); when one of these hosts becomes unavailable, another host in the redundancy group will take over, with no loss of network traffic. CARP also allows a degree of load sharing between systems.

Although creating redundant firewalls is one of its most common uses, CARP isn't a firewall-specific protocol. It can be used to ensure service continuity and/or load sharing to a number of network services.

Anecdotally, the OpenBSD team wanted to produce a free implementation of the IETF standard protocols, VRRP (Virtual Router Redundancy Protocol), defined in [RFC3768], and HSRP (Hot Standby Router Protocol), defined in [RFC2281]; but Cisco, claiming patent rights on it, firmly informed the OpenBSD community that Cisco would defend its patents for VRRP implementation (see [CARP] for more details), thus forcing the OpenBSD developers to create a new, competing protocol designed to be fundamentally different from VRRP. And the funny thing is, putting CARP hosts on a network with Cisco VRRP hosts made Cisco routers crash [LUCAS].

CARP is a multicast protocol, grouping several physical computers together under one or more virtual addresses. Of these, one system is the master and responds to all packets destined for the group; the other systems (backups) just stand by, waiting for any problem to take its place (as it happens between co-workers...).

At configurable intervals, the master advertises its operation on IP protocol number 112. If the master goes offline, the other hosts in the redundancy group begin to advertise. The host that's able to advertise most frequently becomes the new master. When the main system comes back up, it becomes a backup host by default, although it can be configured to try to become master again.

As you can see, CARP only creates and manages the virtual network interface; it's up to the system administrator to synchronize data between applications, using pfsync(4) (which we'll discuss in the next chapter), rsync or whatever protocol is appropriate for the specific application.

4.1 Configuration parameters

CARP configuration is done via the sysctl(8) and ifconfig(8) commands. There are three relevant sysctl(2) variables:

net.inet.carp.allow

defines whether the host handles CARP packets or not. It is enabled by default;

net.inet.carp.log

defines whether to log CARP messages or not. It may be a value between 0 and 7, corresponding to the syslog(3) priorities, and defaults to 2 (i.e. only CARP state changes are logged);

net.inet.carp.preempt

if set to 0 (default), the host won't try to become master if it receives CARP advertisements from another master. Otherwise, it will try to become master if it is able to advertise more frequently than the current master. This option also enables failing over all interfaces in the event that one interface goes down. In fact, if one physical CARP-enabled interface goes down, CARP will increase by 1 the demotion counter (see below) for all groups that the interface belongs to, thus allowing the election of new masters on all subnets.

The syntax for configuring CARP with ifconfig(8) is:

    ifconfig carpN create

    ifconfig carpN [advbase n] [advskew n] [balancing mode]   \
    [carpnodes vhid:advskew,vhid:advskew,...] [carpdev iface] \
    [[-]carppeer peer_address] [pass passphrase] [state state] [vhid host-id]

carpN

the name of the carp(4) virtual interface.

advbase, advskew

these values determine the interval between two consecutive CARP advertisements. This interval (in seconds) is given by the formula (advbase + (advskew / 255)); increasing advbase will decrease network traffic but increase the delay in electing the new master. Small advskew values allow a host to advertise more frequently, increasing its probability to become master. The values of advbase and advskew must be in the range 0 to 254 and default to 1 and 0 respectively;

balancing

sets the load balancing mode (which will be discussed later); valid modes are ip, ip-stealth and ip-unicast;

carpnodes

a comma-separated list of vhid:advskew pairs that actually define how the load should be shared among the configured carp nodes (see below for further details);

carpdev

specifies the physical interface that belongs to this redundancy group. By default, CARP uses the physical interface on the same subnet as the virtual interface;

[-]carppeer

allows you to specify the IP address of the other CARP peer(s), instead of using the default multicast group; this allows to use ipsec(4) to protect carp(4) traffic;

pass

the authentication password to use when talking to other CARP-enabled hosts in the redundancy group. This must be the same on all members of the group;

state

force a carp(4) interface into a specific state (init, backup or master);

vhid

the Virtual Host ID. This is a unique number (between 1 and 255) that is used to identify the redundancy group to the other nodes on the network.

4.1.1 The demotion counter

Besides basic configuration, the ifconfig(8) command also allows you to tweak the CARP demotion counter, which is a measure of how "ready" a host is to become master of a CARP group [CARPFAQ] (the higher the counter, the less ready the host). Let's see it in more detail.

CARP interfaces are divided in groups (by default all carp(4) interfaces are members of the "carp" interface group) and each group is assigned a demotion counter, whose value can be viewed by running the following command:

$ ifconfig -g carp
carp: carp demote count 0

The demotion counter comes in handy mainly when:

• you want to momentarily prevent a host from becoming master: for instance, at boot time, the rc(8) script increases the demotion counter by 128 before starting the network and decreases it by the same amount once all interfaces have been initialized and all network daemons have been started (the demotion counter can't be set to an absolute value, but only increased or decreased by a certain amount):
  /etc/rc

  ifconfig -g carp carpdemote 128
  [ ... ]
  ifconfig -g carp -carpdemote 128
  

• you want to gracefully failover only a limited number of a host's carp(4) interfaces (not all of them, as it happens when an interface goes down and preempt is enabled). In the following example, we will failover the carp1 and carp2 interfaces and leave the state of the others unchanged:

  # ifconfig carp1 group morituri
  # ifconfig carp2 group morituri
  # ifconfig morituri
  carp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
          carp: MASTER carpdev sis0 vhid 1 advbase 1 advskew 100
          groups: carp morituri
          inet 1.2.3.4 netmask 0xffffff00 broadcast 1.2.3.255
  carp2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
          carp: MASTER carpdev sis1 vhid 2 advbase 1 advskew 100
          groups: carp morituri
          inet 2.3.4.5 netmask 0xffffff00 broadcast 2.3.4.255 
  # ifconfig -g morituri
  morituri: carp demote count 0
  # ifconfig -g morituri carpdemote 50
  # ifconfig -g morituri
  morituri: carp demote count 50
  

For further details on the CARP demotion counter, please refer to [CARPFAQ].

4.1.2 Load balancing

CARP allows you to load balance incoming network traffic among a set of CARP-enabled hosts; first, you need to create a load balancing group by configuring, on each balanced carp(4) interface, as many VHIDs as hosts in the balancing group; the advskew on each VHID will be configured so that each host will be the master on a separate VHID (see below for a practical example).

Since balancing requires that all CARP hosts receive network traffic destined to the CARP address, the virtual interface will use a multicast MAC address, forcing the switch to send incoming traffic to all nodes in the redundancy group.

CARP uses the hash of the source and destination addresses of the IP packet to determine which VHID (and therefore which host) should accept the packet; balancing can be enabled using ifconfig(8), by setting the balancing option to "ip". For example:

# ifconfig carp0 balancing ip carpnodes 1:0,2:100

Alternatively, you can set the balancing option to "ip-stealth" (stealth mode), in order to prevent hosts from sending packets with their virtual MAC address as source; this will prevent the switch from learning the virtual MAC address, forcing it to flood the traffic to all its ports. Last, if you're using a hub or a switch that supports some kind of monitoring mode, you can set balancing to "ip-unicast".

4.2 Parameters configuration

Now it's time to configure CARP on our firewalls. To examine two slightly different CARP configurations, we will set up the two internal firewalls (Mickey and Minnie, between LAN and DMZ) in active/stand-by mode, with only one system filtering the whole network traffic and the other one acting as a hot spare; the two external firewalls (Donald and Daisy, separating the DMZ from the internet), instead, will be in active/active mode, sharing the traffic load.

So let's recap the firewalls adresses, as we have seen them in the network diagram:

4.2.1 Active/standby configuration

Let's start with Mickey and Minnie: first, we need to create the carp* devices and configure them with ifconfig(8):

mickey# ifconfig carp0 172.16.0.202/24 vhid 1 pass password1 advbase 1 advskew 0
mickey# ifconfig carp1 172.16.240.202/24 vhid 2 pass password2 advbase 1 advskew 0

minnie# ifconfig carp0 172.16.0.202/24 vhid 1 pass password1 advbase 1 advskew 100
minnie# ifconfig carp1 172.16.240.202/24 vhid 2 pass password2 advbase 1 advskew 100

We have just created the interfaces, assigned them an IP address, a virtual host ID (1 on the LAN, 2 on the DMZ) and a password (probably not the most secure...) for authentication. We also decided that, whenever possible, Mickey will be the master; this is done by giving Minnie a higher advskew value (100), thus making the interval between its advertisements (1 + 100 / 255) higher than the interval between Mickey's advertisements (1 + 0 / 255). And, as we've seen above, the host that's able to advertise most frequently becomes master.

Furthermore, by setting net.inet.carp.preempt to "1" on Mickey, we ensure that Mickey will always try to become the master:

mickey# sysctl net.inet.carp.preempt=1
net.inet.carp.preempt: 0 -> 1

To make these settings permanent after reboot, we just need to edit the /etc/hostname.carp* and /etc/sysctl.conf files on Mickey:

inet 172.16.0.202 255.255.255.0 172.16.0.255 vhid 1 pass password1 advbase 1 advskew 0

inet 172.16.240.202 255.255.255.0 172.16.240.255 vhid 2 pass password2 advbase 1 advskew 0

[...]
net.inet.carp.preempt=1

and on Minnie:

inet 172.16.0.202 255.255.255.0 172.16.0.255 vhid 1 pass password1 advbase 1 advskew 100

inet 172.16.240.202 255.255.255.0 172.16.240.255 vhid 2 pass password2 advbase 1 advskew 100

Note: to make the adoption of CARP easier on pre-existing networks, CARP allows using the physical address of a host as the virtual address of the whole redundancy group.

4.2.2 Active/active configuration

Now let's get on to Donald and Daisy and start by configuring their DMZ interfaces. As before, we will create the carp0 device on each machine, but this time, to enable load balancing, we will use the carpnodes option to assign two different Virtual Host IDs to the interface (VHIDs 3 and 4).

On VHID 3, we will set the advskew of Donald and Daisy to 0 and 100 respectively: this will ensure that Donald becomes master for that VHID; on VHID 4, instead, we will do the opposite, by setting the advskew of Donald and Daisy to 100 and 0 respectively, in order to force Daisy to become master for VHID 4:

donald# ifconfig carp0 172.16.240.102/24 balancing ip carpnodes 3:0,4:100 \
> pass password3
donald# sysctl net.inet.carp.preempt=1
net.inet.carp.preempt: 0 -> 1

daisy# ifconfig carp0 172.16.240.102/24 balancing ip carpnodes 3:100,4:0 \
> pass password3
daisy# sysctl net.inet.carp.preempt=1
net.inet.carp.preempt: 0 -> 1

We now have two redundancy groups with the same IP address, but each with a different master:

donald# ifconfig carp0
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr 01:00:5e:00:01:01
        carp: carpdev rl1 advbase 1 balancing ip
                state MASTER vhid 3 advskew 0
                state BACKUP vhid 4 advskew 100
        groups: carp
        inet 172.16.240.102 netmask 0xffffff00 broadcast 172.16.240.255
        inet6 fe80::2c0:a8ff:fe8e:b112%carp0 prefixlen 64 scopeid 0x5

daisy# ifconfig carp0
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr 01:00:5e:00:01:01
        carp: carpdev rl1 advbase 1 balancing ip
                state BACKUP vhid 3 advskew 100
                state MASTER vhid 4 advskew 0
        groups: carp
        inet 172.16.240.102 netmask 0xffffff00 broadcast 172.16.240.255
        inet6 fe80::219:d2ff:fe02:6469%carp0 prefixlen 64 scopeid 0x5

To make these settings permanent across reboots, we need to edit the startup files on Donald:

inet 172.16.240.102 255.255.255.0 172.16.240.255 balancing ip carpnodes 3:0,4:100 pass password3

[...]
net.inet.carp.preempt=1

and Daisy:

inet 172.16.240.102 255.255.255.0 172.16.240.255 balancing ip carpnodes 3:100,4:0 pass password3

[...]
net.inet.carp.preempt=1

Now we just have to do the same on the external network interfaces, with another two Virtual Host IDs (VHIDs 5 and 6):

donald# ifconfig carp1 172.16.250.102/24 balancing ip carpnodes 5:0,6:100 \
> pass password5

daisy# ifconfig carp1 172.16.250.102/24 balancing ip carpnodes 5:100,6:0 \
> pass password5

and edit the startup files on Donald:

inet 172.16.250.102 255.255.255.0 172.16.250.255 balancing ip carpnodes 5:0,6:100 pass password5

and Daisy:

inet 172.16.250.102 255.255.255.0 172.16.250.255 balancing ip carpnodes 5:100,6:0 pass password5

Though the above configuration involves only a couple of machines, it can be easily extended to up to 32 hosts. One last note: load sharing won't probably achieve a perfect 50/50 distribution between the two machines, since CARP uses a hash of the source and destination IP addresses to determine which system should accept a packet, not the actual load.