Cryptosystems: Debugging IPSec

In the last article, we configured the files necessary to negotiate an IPSec VPN using racoon. I have a lot of troubleshooting scenarios
to cover this week, so let's carry on where we left off.

The one configuration I purposefully left out last time is the necessary
changes to the firewall ruleset. You might remember that we need to allow UDP
port 500 for IKE and protocol number 50 for ESP. The syntax for those rules
will depend upon whether your system is protected by ipfw,
ipfilter, or is behind another type of firewall. It will also
depend upon the order of the existing rules in your ruleset and the degree of
paranoia dictated by your security policy.

For now, create two rules near the top of your ruleset. We may have to
tweak their placement later on when we start working with the negotiated
tunnel. Here is an example of some rules to an ipfw ruleset:

#rules to allow IPSec VPN
add 00201 allow log esp from any to any
add 00203 allow log udp from any 500 to any

Until you are happy with your tunnel, you should use the log
word. If your security policy allows it, consider starting with the
any keyword until your tunnel is successful. Afterwards, you can
tighten up those rules by specifying particular sources, destinations,
directions, and interfaces.

Here is an example of some rules to an ipfilter ruleset:

#rules to allow IPSec VPN
pass in log quick proto esp from any to any
pass in log quick proto udp from any port = 500 to any port = 500

Post-Configuration Tests

Once I've added my firewall rules, I'll reboot into the new IPSec enabled
kernel. I'll also watch the startup messages as they go by. In particular, I'm
looking for error messages. If you see something interesting in your startup
messages, press the scroll lock key and use your page up key to go back to the
error. Make sure all of your firewall rules load successfully. If you have a
typo, your startup message will indicate which rule prevented the rulebase from
loading.

You should also see this message in your output:

ipsec: enabled

If you don't, doublecheck that /etc/rc.conf contains this
line:

ipsec_enable="YES"

You should also see your route being added when racoon
initializes:

Starting local daemons:add net 192.168.1.0: gateway
10.0.0.1

If you get this error:

racoon: failed to parse configuration file.

you have a typo in /usr/local/etc/racoon/racoon.conf. Check
carefully for missing semicolons or braces and typos in the keywords you added.
Once you've made your change, do a shutdown now and re-check for
error messages. You want to be able to boot without error messages.

Once you have resolved any error messages, login and verify that
racoon has indeed started:

If you don't have a policy, there is a problem with
/etc/ipsec.conf. You either have a typo or you forgot to tell
/etc/rc.conf to load that file at bootup. Once racoon
is up at both peers, and both peers have a matching policy, you are ready to try
tunnel negotiation.