One of the fundamental constructs of IPsec is the Security Association, or SA. According to RFC 2401, a "Security Association is a simplex 'connection' that affords security services to the traffic carried by it." SAs provide security services by using either AH or ESP, but not both (if a traffic stream uses both AH and ESP, it has two--or more--SAs). For typical IP traffic, there will be two SAs: one in each direction that traffic flows (one each for source and destination host).

An SA is identified by three things:

A Security Parameter Index (SPI)

An IP destination address

A security protocol (AH or ESP) identifier

IPsec defines two modes for exchanging secured data, tunnel mode and transport mode. IPsec transport mode protects upper-layer protocols, and is used between end-nodes. This approach allows end-to-end security, because the host originating the packet is also securing it and the destination host is able to verify the security, either by decrypting the packet or certifying the authentication.

Tunnel mode IPsec protects the entire contents of the tunneled packets. The tunneled packets are accepted by a system acting as a security gateway, encapsulated inside a set of IPsec/IP headers, and forwarded to the other end of the tunnel, where the original packets are extracted (after being certified or decrypted) and then passed along to their ultimate destination.

The packets are only secured as long as they are "inside" the tunnel, although the originating and destination hosts could be sending secured packets themselves, so that the tunnel systems are encapsulating packets that have already been secured.

Transport mode is good for any two individual hosts that want to communicate securely; tunnel mode is the foundation of the Virtual Private Network or VPN. Tunnel mode is also required any time a security gateway (a device offering IPsec services to other systems) is involved at either end of an IPsec transmission. Two security gateways must always communicate by tunneling IP packets inside IPsec packets; the same goes for an individual host communicating with a security gateway. This occurs any time a mobile laptop user logs into a corporate VPN from the road, for example.

Data origin authentication and connectionless integrity, through use of authentication mechanisms such as cryptographic message hashes

An anti-replay service (the same as used by the Authentication Header)

Limited traffic flow confidentiality, when used in tunnel mode

ESP and AH authentication services are slightly different: ESP authentication services are ordinarily provided only on the packet payload, while AH authenticates almost the entire packet including headers.

AH is specified in RFC 2402, and the header is shown in the figure below (taken from RFC 2402).

The Sequence Number field is mandatory for all AH and ESP headers, and is used to provide anti-replay services. Every time a new packet is sent, the Sequence Number is increased by one (the first packet sent with a given SA will have a Sequence Number of 1). When the receiving host elects to use the anti-replay service for a particular SA, the host checks the Sequence Number: if it receives a packet with a Sequence Number value that it has already received, that packet is discarded.

The authentication data field contains whatever data is required by the authentication mechanisms specified for that particular SA to authenticate the packet. This value is called an Integrity Check Value or (ICV), it may contain a keyed Message Authentication Code (MAC) based on a symmetric encryption algorithm (such as CAST or Triple-DES) or a one-way hash function such as MD5 or SHA-1.

ESP is specified in RFC 2406, and while similar to AH in many ways it provides a wider selection of security services and can be a bit more complex.

The ESP header format is shown in the figure below (taken from RFC 2406).

The most obvious difference between ESP and AH is that the ESP header's Next Header field appears at the end of the security payload. Of course, since the header may be encapsulating an encrypted payload, you don't need to know what header to expect next until after you've decrypted the payload. Thus, the ESP Next Header field is placed after, rather than before, the payload. ESP's authentication service covers only the payload itself, not the IP headers of its own packet as with the Authentication Header. And the confidentiality service covers only the payload itself; obviously, you can't encrypt the IP headers of the packet intended to deliver the payload and still expect any intermediate routers to be able to process the packet. Of course, if you're using tunneling, you can encrypt everything, but only everything in the tunneled packet itself.