IPSec Certificate Basics

So you've got IPSec running between your hosts using preshared
keys, and you want to move to X.509 certificates? It's a good idea.
They're easier to manage than shared keys, and preshared keys may
disappear in IPSec version 2.

It's a relatively easy move, but if you're not familiar with
OpenSSL, it can get frustrating. This article will ease that
frustration by providing a step-by-step guide to deploying a
certificate-based authentication scheme. The article assumes you are
configuring certificates for use between various hosts and a tunnel
server, but the functionality and setup are identical for transport
mode as well.

Quick review

Let's just run down the terms we'll be using here. X.509
certificates are based on public/private keypairs. Each certificate
contains a public key, along with other information (identifying and
not), such as the owner's common name and the certificate's expiration
date. The owner keeps her private key in a separate file.

The certs are signed by a Certificate Authority, or CA, and
contain information as to which authority signed it. This digitally
proves that the certificate is authentic and that the information
contained within it is accurate--perfect for verifying the identity
of a remote host. The CA's authenticity is verified by its
certificate, which is generally available to the public.

Requirements

Running certificate-based IPSec authentication requires two
things. First off is a recent version of racoon; grab the latest
available in /usr/ports/security/racoon (currently
20011215a). Next is OpenSSL, version 0.9.5a or higher. The version
supplied with FreeBSD by default does not have the useful CA.pl
script, so you should download and install the latest version, which
will put CA.pl in /usr/local/ssl/misc/CA.pl
by default.

Creating your own CA

With OpenSSL, you can create your own certificates and even your
own Certificate Authority--meaning that you can create, sign, and
distribute certificates to the world. While the rest of the world may
not recognize your standing as a CA ("Do I want to accept a
certificate signed by Joe's Garage and Certificate Authority? No!"),
at least you will recognize yourself as one. And that's all you need
for hosts connecting to your tunnel server.

You may want to create your CA on the tunnel server, or you may not. It's up to you.

To start, login to whatever host you decide upon as your CA, and create a
directory where you'll manage your certificates. You'll then create
your CA subdirectory (demoCA) by running:

/usr/local/ssl/misc/CA.pl -newca

When prompted for the CA certificate filename, press Enter. You'll
then be prompted for a password to protect the CA's private key. It's very important that the password stays safe, or anyone could sign
certificates as you, so make it a good one. Next, you will be
prompted to enter identifying information about your location,
company, common name, and email address. It's all self-explanatory,
except for the common name. This is a mandatory bit of uniquely
identifying data, such as your host's FQDN or your name.

After entering the required information, the demoCA
subdirectory is created. If you're paranoid, you may want to chmod
the private key file (demoCA/private/cakey.pem) so that
only root can read it. But this shouldn't be necessary since you
used a really good password to protect it, right?

You can ignore most of demoCA's remaining contents,
but you'll need to use demoCA/cacert.pem in the near
future.

Creating and installing certificates

Now that you're set up to sign certificates, you can begin
creating them for all the hosts using IPSec on your network. There are a few simple steps, repeated for each of the remote boxes and the
tunnel server.

The first step is to create a public key and certificate request.
On the CA server, run:

Next, the certificate must be signed. It's easiest to do this with
the CA.pl -sign command. Give the CA's password, then
respond with y to the prompts, indicating that you want to sign the certificate.

You'll now have three files to copy to the appropriate machine:
demoCA/cacert.pem, newcert.pem, and
privkey.pem. Put them in /usr/local/etc/racoon/cert on the client you are configuring.
Make sure the directory is only readable by root; otherwise, malicious
parties could steal your private key and masquerade as you!

The final step is to allow the host to recognize other
certificates signed by your CA. Login to the machine and run:

Make sure to replace cacert.pem with whatever you named
the file. The symbolic link allows OpenSSL to find your CA's
certificate based on its hash, thus verifying that the
certificate is authentic and acceptable.

Repeat the above steps for all hosts that you want to have connect
to your tunnel server.

Next, you'll need to change the remote block(s) to use and accept
certificates. The necessary changes are bolded below for IPSec over
IPv4 on anonymous connections. The changes are the same for IPv6 and
identified connections.

my_identifier asn1dn tells racoon that the local host
should identify itself by using its Distinguished Name (DN) as
defined by its certificate (that is, all of the fields in its
certificate strung together). peers_identifier asn1dn
states that the remote host should identify itself with its
certificate's DN. verify_identifier on ensures that the
identification the peer sends is the same type that is specified in
peers_identifier; in this case, it's asn1dn. The
certificate_type line tells racoon to use an X.509
certificate. cert.pem is the name of the local host's
certificate file, while key.pem is its private key.
These file names should be changed to whatever you named them.

The last line to change, authentication_method rsasig,
tells racoon that it should use certificate-based identification of
its peers.

Once you've made the necessary changes on each host's racoon.conf,
you're ready to go. The next section discusses how to proceed and
gives some troubleshooting tips.

Running and troubleshooting

Kill the racoon daemon on your server and client. Restart it, and
tail your logfile. Ping one host from the other. You should see lines
similar to:

If you don't, there are a few things to check. First, make sure your path and file names are specified correctly in racoon.conf. Next, check that you created a symbolic link, mentioned above in "Creating and installing certificates." Double-check your configuration, making sure your identifier and your
peer's are set correctly, along with any other lines you changed
(certificate related or not.) Finally, check your security policies
and make sure they still match.

Conclusion

In this article, you learned all the basics necessary to use X.509
certificates for authentication in IPSec on a FreeBSD box. You're now
ready to enjoy the benefits, and you don't have to worry about creating, changing, and remembering all of your shared keys. Further, since the IETF may discontinue preshared keys in the next version of IPSec, it's one less headache if you upgrade (once IPSec version 2 comes out, of course).

Mike DeGraw-Bertsch
is a security and Unix system administration
consultant in the Boston, Mass. area. When he's not at a job, writing,
hacking with Perl, or playing with his wireless network, he can usually be
found playing goal in ice hockey.