You should always maintain secure communications within your network and between
networks to whatever degree possible. Internet Protocol Security (IPSec)
allows you to control security using flexible policies to adapt to any network.
IPSec secures data in the private and public environments of your network by
providing a strong cryptography-based defense.

In this chapter, we discuss securing your network with IPSec, which includes
the following:

Planning an IPSec deployment

Configuring IPSec policies

Deploying and managing IPSec policies

Planning an IPSec Deployment

One of the best features of IPSec is its flexibility. Unfortunately,
that's a double-edged sword when it comes to network design. IPSec can be
configured for the needs of almost any network, but many decisions must be made
regarding its configuration. Some decisions are made for you by the way in which
you are using IPSec or by the types of client operating systems that you are
running on your network; other decisions are made by you. We now discuss the
steps that you must take to deploy IPSec, which include the following:

Deciding which IPSec mode to use

Planning authentication methods for IPSec

Securing authentication with IPSec

Testing the functionality of existing applications and services

Deciding Which IPSec Mode to Use

This decision is rather easy because it has already been made for you based
on the way you are using IPSec. Basically, two different types of communications
exist that you need to securecommunications within a network and
communications between networks. You choose one of two modes of IPSec
securitytransport mode or tunnel modebased on the type of
communication you need to make secure.

Transport Mode

You can use IPSec in transport mode to secure communications between
two computers on the same network. This can be server-to-server or server-to-client
communications. Once configured, IPSec provides end-to-end security based on
the authentication and encryption settings that you apply. Figure
3.1 illustrates the concept of IPSec transport mode.

Tunnel Mode

You can use IPSec in tunnel mode to secure communications between two
networks. You define the endpoints of the tunnel, and the system maintains a
secure connection between the endpoints. The endpoints are generally router
interfaces. You can specify the types of encryption and authentication protocols
that will be required for all traffic. Figure
3.2 illustrates the concept of IPSec tunnel mode.

Planning Authentication Methods for IPSec

Authentication is the process of a network entity proving its identity
in a confidential manner. Because authentication must be confidential, computers
need to be able to authenticate to each other using a secure method. IPSec uses
three main methods of authentication. You make your choice based on the
capabilities of your servers and clients and the requirements of your
organization. The three main methods from which you can choose are as
follows:

Kerberos

Certificates

Preshared key

Kerberos

Kerberos is the default authentication method for Windows 2000 Server and
Windows Server 2003. With Kerberos, the client must prove its identity to the
server and the server must also prove its identity to the client. This is
referred to as mutual authentication. Kerberos can only be used with
Windows 2000 Professional and later clients. You should use Kerberos when all of
your clients are in the same domain or Kerberos realm and can authenticate using
Kerberos and when you want to use a method that requires the least
administrative effort.

Figure 3.3 illustrates that Kerberos
is the default protocol in the IP security policy.

CAUTION

When an exam question specifies an authentication method that requires the
least administrative effort, and all of the clients and servers are Windows
2000, Windows Server 2003, or Windows XP Professional, you should first
determine whether all of the clients are in the same domain. If so, you should
use the default Kerberos authentication method. If not, then other methods will
have to be considered.

Certificates

As we discussed earlier, certificates are a method of granting access to a
user based on the user's unique identification and whether the user possesses
the right keys or algorithms to "unlock" the appropriate doors. You
issue a certificate to a computer as a means for it to hold the key and as a
means for you to track that the computer has the key. You can stipulate in an
IPSec policy, as shown in Figure 3.4,
that the computer or user must use a certificate (Secure Server) or that it
should use a certificate if it has one (Server).

Figure 3.4 illustrates the default policies
that are included in Windows Server 2003. Microsoft recommends that you create
your own policies by modifying the default polices or by starting from scratch
and using the Create IP Security Rule Wizard. Depending on your settings, you
will create a policy that must be met in its entirety for communications to
continue or a policy that can be negotiated between computers.

Preshared Key

A preshared key is a string that you can use to authenticate
computers as a last resort. You should not use a preshared key if any other
authentication method is available. A preshared key is a symmetric key, which
means that the same key that is used to encrypt can also be used to decrypt. You
should configure the same string on all computers that you want to authenticate.

The main problem with a preshared key is that you have no way of knowing whether
the key is discovered. Also, the key is not specific to any individual. Therefore,
an attacker could use the key to authenticate to a network and you could not
trace the attack back to an individual. In addition, the preshared key is stored
in the Registry in plain text form. It can be configured from the Authentication
Methods tab in the properties of an IPSec policy as shown in Figure
3.5.

Figure
3.4 You can require or request certificates as part of an IPSec policy.

Figure
3.5 You can use a string to configure a preshared key when no other authentication
options exist.

Security Authentication with IPSec

As mentioned previously, your goal is to allow computers to authenticate to
each other without anyone else being able to see the process. To facilitate
this, the authentication process must be encrypted. Two main protocols can be
used when authenticating with IPSec. These are Secure Hash Algorithm (SHA) and
Message Digest 5 (MD5). In addition, after authentication takes place, you can
use multiple levels of encryption for the data itself. You should be familiar
with the following terms and settings for IPSec encryption. These settings are
illustrated in Figure 3.6 and can be
located by clicking the General tab of an IPSec policy, then the Settings button,
and finally the Methods button.

Secure Hash Algorithm (SHA)

Message Digest 5 (MD5)

Data Encryption Standard (DES)

Triple DES (3DES)

Figure
3.6 You can choose the type of encryption used by IPSec during authentication
and during data transfer.

Secure Hash Algorithm (SHA)

Secure Hash Algorithm (SHA) is the accepted standard for securing
authentication of computers working with government contracts. It is used as
part of the Federal Information Processing Standard (FIPS). SHA is a very
high-security method that uses a 160-bit encryption key.

Message Digest 5 (MD5)

Message Digest 5 (MD5) is used for most commercial applications. You can use
MD5 to secure authentication as well as data. This high-security method uses a
128-bit encryption key. It also has a lower performance overhead than that of
SHA.

Data Encryption Standard (DES)

Data Encryption Standard (DES) is the lowest encryption strength of the
Diffie-Hellman algorithms. It produces only a 56-bit key and is therefore not
recommended for use in a high-security environment.

Triple DES (3DES)

Triple DES (3DES) is a much stronger Diffie-Hellman algorithm than DES, and
produces a 168 to 2048-bit key. It is recommended for use in medium- to
high-security networks.

Testing the Functionality of Existing Applications and Services

Remember, the reason that you originally decided to use security methods was
to protect the integrity and the productivity of the network. For this reason,
you should always test the applications and services that you are running on
your network to make sure that they can still function with the IPSec rules that
you have configured. There are many reasons that an application could cease to
function or function with errors. Most of these involve the fact that IPSec
rules can be used to filter traffic. Sometimes these filters, as indicated in
the following list, can cause unexpected results:

Filtered ports

Filtered IP addresses

Filtered protocols

You can use the IP Filter Wizard (see Figure
3.7) to create multiple filters in an IPSec policy rule. If you use more
than one filter in a single IPSec policy rule, be aware that the order that
the filters are processed in is not necessarily the order in which you are viewing
them. Instead, the IPSec Policy Agent reads the policy, and the filters are
processed into one ordered list that is sorted from the most to the least specific.
You can use the IPSec Monitor console to view the filters sorted by their weight.
If you change or delete a filter, the IPSec Policy Agent reorders the filters
based on what remains. Because of this, you should always test applications
that use IPSec after applying IPSec and after changing any filters. You should
use a test lab, if one is available, to test the effect of IPSec rules before
assigning them in a production environment.

Figure
3.7 You can use the IP Filter Wizard to create multiple filters in an IPSec
policy rule.

CAUTION

IPSec can now function through some Network Address Translation (NAT)
infrastructures. As long as the version of NAT is compliant with the latest RFC
and is configured to allow User Datagram Protocol (UDP) traffic, the Internet
Key Exchange (IKE) protocol will detect the presence of NAT and use UDP-ESP
encapsulation to allow the traffic to pass through.