RFC 2401 defines the architecture for IPSec, including the framework and the services provided. RFC 2401 also defines how the services work together and how and where to use them.

IPSec uses three main protocols to create a security framework:

Internet Key Exchange (IKE)

Encapsulating Security Payload (ESP)

Authentication Header (AH)

Internet Key Exchange (IKE) is the protocol for exchanging keys. It provides a framework for the negotiation of security parameters and establishes authenticated keys.

IPSec uses symmetrical encryption algorithms for data protection, which are more efficient and easier to implement in hardware than other types of algorithms.

These algorithms need a secure method of key exchange to ensure data protection. The IKE protocols provide the capability for secure key exchange.

Internet Key Exchange (IKE) solves the problems of manual and unscalable implementation of IPSec by automating the entire key exchange process, including

negotiation of security association (SA) characteristics

automatic key generation

automatic key refresh

manageable manual configuration

To implement a VPN solution with encryption, the periodic changing of encryption keys is necessary. Failure to change these keys makes the network susceptible to brute force attacks.

IPSec solves this problem with the IKE protocol, which uses two other protocols to authenticate a peer and generate keys. The IKE protocol uses a mathematical routine called a Diffie-Hellman exchange to generate symmetrical keys to be used by two IPSec peers.

IKE also manages the negotiation of other security parameters, such as data to be protected, strength of the keys, hash methods used, and whether packets are protected from replay. IKE uses UDP port 500.

IKE negotiates an SA, which is an agreement between two peers engaging in an IPSec exchange. To establish successful communication, the SA comprises three required parameters:

Oakley

ISAKMP

Skeme

Resume of the IKE Phases

IKE Phase 1:

* Authenticate the peers

Optionally, phase 1 can also include an authentication in which each peer is able to verify the identity of the other. This conversation between two IPSec peers can be subject to eavesdropping with no significant vulnerability of the keys being recovered.

* Negotiate a bidirectional SA

Phase 1 SAs are bidirectional – data may be sent and received using the same key material generated. Two modes are available for phase 1 SA negotiations: main mode or aggressive mode.

* Main mode (6 messages) or aggressive mode (3 messages)

in – Main mode

In the main mode, an IKE session begins with the initiator sending a proposal or proposals to the responder. These proposals define which encryption and authentication protocols are acceptable, how long keys should remain active, and whether perfect forward secrecy should be enforced. Multiple proposals can be sent in one offering.

The first exchange between nodes establishes the basic security policy. The responder chooses the appropriate proposal and sends it to the initiator. The next exchange passes Diffie-Hellman public keys and other data. All further negotiation is encrypted within the IKE SA. The third exchange authenticates the ISAKMP session. Once the IKE SA is established, IPSec negotiation (quick mode) begins.

in – Aggressive mode

The aggressive mode squeezes the IKE SA negotiation into three packets, with all data required for the SA passed by the initiator. The responder sends the proposal, key material, and identification, and authenticates the session in the next packet. The initiator replies by authenticating the session. Negotiation is quicker, and the initiator and responder ID pass in plaintext.

** show crypto isakmp sa

IKE Phase 1 is the initial negotiation of SAs between two IPSec peers.

IKE Phase 1.5 (Optional):

* Xauth (User Authentication)

IKE Phase 1.5 is optional. To further authenticate VPN participants (clients), you can use a protocol called Extended Authentication (Xauth), which provides user authentication of IPSec tunnels within the IKE protocol. Additionally, you can exchange other parameters between the peers.

* Mode config (Push Config)

Mode configuration is used to deliver parameters such as IP address and DNS address to the client.

IKE Phase 2:

* IPsec SAs/SPIs

IKE Phase 2 SAs are negotiated by the IKE process (ISAKMP) on behalf of other services such as IPSec, which need key material for operation.

Because the SAs used by IPSec are unidirectional, separate key exchanges are needed for data flowing in the forward direction and the reverse direction. The two peers have already agreed upon the transform sets, hash methods, and other parameters during the phase 1 negotiation.

* Quick mode

Quick mode is the method used for the phase 2 SA negotiations. – The quick mode IPSec negotiation is similar to an aggressive mode IKE negotiation, except negotiation must be protected within an IKE SA. Quick mode negotiates the SA for the data encryption and manages the key exchange for that IPSec SA.

** show crypto ipsec sa

Main mode:

has three two-way exchanges between the initiator and receiver:

— First exchange—The algorithms and hashes used to secure the IKE communications are negotiated and agreed upon between peers.

— Second exchange—This exchange uses a DH exchange to generate shared secret keys and pass nonces, which are random numbers sent to the other party, signed, and returned to prove their identity. The shared secret key is used to generate all the other encryption and authentication keys.

— Third exchange—This exchange verifies the other side’s identity. It is used to authenticate the remote peer. The main outcome of main mode is a secure communication path for subsequent exchanges between the peers. Without proper authentication, you might establish a secure communication channel with a hacker who could be stealing all your sensitive material.

Aggressive mode:

fewer exchanges are done and with fewer packets. The first exchange covers almost all of the steps:

the IKE policy set negotiation; the DH public key generation; a nonce, which the other party signs; and an identity packet, which can be used to verify the identity of the other party via a third party. The receiver sends everything back that is needed to complete the exchange. The only thing left is for the initiator to confirm the exchange.

IPSec solves the encryption problem of a VPN solution by using the IKE protocol, which uses UDP port 500. IKE is executed in two phases. Phase 1 initializes the SAs between two peers, and phase 2 manages the bidirectional flow of information between the two peers. Phase 1 can be split into two, so that you can do an extended authentication (Xauth) on VPN clients. IKE can be used in a combination of three modes – main mode, aggressive mode, and quick mode.