IPsec configuration on OpenBSD is a pretty easy and straightforward process, especially compared to most other implementations; nevertheless, IPsec is a rather complicated beast and a good working knowledge of its protocols and internals is essential to configure it and get it to work properly. Therefore, before beginning the configuration, let's take a brief tour of the IPsec protocols and features.
IPsec (IP security) is a suite of standard protocols designed to provide interoperable, high quality, cryptographically-based security
[RFC4301] for protecting communications over IPv4 and IPv6 networks. The main security services offered by IPsec are:
These security services are provided at the IP layer (layer 3 of the OSI model), thus protecting all protocols that may be carried over IP, including IP itself.
Most of IPsec security services are provided using two traffic security protocols:
ESP is by far the most popular of the two protocols, since it provides confidentiality by encrypting network traffic, thus protecting transmitted data from passive attacks. On the other hand, AH provides stronger authentication than ESP as it protects part of the outer IP header as well as the next level protocol data, while ESP only protects the inner (encapsulated) IP header; however, this feature, in addition to not being of great use in most cases, also violates the modularization of the protocol stack (see [SCHNEIER], where the AH protocol is proposed for complete elimination).
AH and ESP may also be applied in combination with each other to exploit the strengths of both protocols but, in most real-world scenarios, ESP alone is enough.
Both ESP and AH support two modes of operation:
The flexibility of tunnel mode allows it to fully supersede the functionality of transport mode, at the reasonable expense of a slightly higher bandwidth overhead. As a consequence, transport mode is rarely used in real-world VPNs and, just like AH, [SCHNEIER] suggests that transport mode be eliminated altogether, with the advantage of significantly reducing IPsec complexity.
In a nutshell, while ESP and tunnel mode are by far the most prevalent choice, AH and transport mode can be considered the black sheeps of the IPsec protocol family!
To actually establish the VPN, the IPsec protocols require that some state data be shared between the VPN endpoints, such as the cryptographic algorithms for encryption and authentication, the keys used as input to the cryptographic algorithms, the current sequence number, the antireplay window and so on.
These data are held in a data structure called a Security Association (SA); SAs are created by a specific protocol, IKEv2 (defined in [RFC4306]), which also has the responsibility of mutually authenticating the two communicating parties, setting up the encrypted channel for secure information exchange (these steps are part of the so-called IKE phase 1) and negotiating the shared secret from which cryptographic keys are derived (IKE phase 2).
A Security Association applies to a single protocol (AH or ESP) and to a single direction of traffic flow; therefore, to secure typical, bi-directional communication between two IPsec-enabled systems, a pair of SAs (one in each direction) is required. IKE explicitly creates SA pairs in recognition of this common usage requirement
[RFC4301].
SAs are collected in a Security Association Database (SAD), where they are uniquely identified by the combination of protocol (AH or ESP), destination address and an arbitrary 32-bit value called the Security Parameter Index (SPI). The SPI has the specific task of helping the receiver to identify the SA under which an incoming packet should be processed.
But how does IPsec decide which datagrams to send through the VPN and which not? For instance, in a typical site-to-site VPN scenario, the IPsec gateway will usually tunnel and/or protect only traffic between the remote LANs, leaving all other traffic unaffected. Well, IPsec makes such decisions based on policies, i.e. user-defined rules stating which packets should be protected using IPsec security services, which should be allowed to bypass IPsec protection and which should be discarded. IPsec policies are applied based on some specific fields in the datagram headers, called selectors, which include: source and destination addresses, Next Layer Protocol, source and destination ports (if used by the next layer protocol).
As with Security Associations, IPsec policies are held in a database, called the Security Policy Database (SPD), which must be consulted during the processing of all traffic (inbound and outbound), including traffic not protected by IPsec, that traverses the IPsec boundary.
To recap, let's have a look at what the (brief) life of an IPsec packet looks like; we will consider the most common case: an ESP tunnel-mode VPN between two remote networks (see picture above). The story begins when the first gateway (GW1) receives an outbound packet from a host (Host1) within its internal network and destined for a host (Host2) on the remote network:
After a brief walk around the Internet, the encapsulated packet hits the second gateway (GW2):
