Strong Firewall in a Routed Xen Dom0

TomEastep

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version
1.2 or any later version published by the Free Software Foundation; with
no Invariant Sections, with no Front-Cover, and with no Back-Cover
Texts. A copy of the license is included in the section entitled
“GNU Free Documentation
License”.

Caution

This article applies to Shorewall 4.0 and later. If you are running
a version of Shorewall earlier than Shorewall 4.0.0 then please see the
documentation for that release.

Before Xen

Prior to adopting Xen, I had a home office crowded with 5 systems,
three monitors a scanner and a printer. The systems were:

Firewall

Public Server in a DMZ (mail)

Private Server (wookie)

My personal Linux Desktop (ursa)

My work system (docked laptop running Windows XP).

The result was a very crowded and noisy room.

After Xen

Xen has allowed me to reduce the noise and clutter considerably. I
now have three systems with two monitors. I've also replaced the
individual printer and scanner with a Multifunction
FAX/Scanner/Printer.

The systems now include:

Combination Firewall/Public Server/Private Server/Wireless
Gateway using Xen (created by building out my Linux desktop system --
Now replaced by a Hewlett-Packard Pavilion a1510y).

My work system.

My Linux desktop (wookie, which is actually the old public
server box)

The Linux systems run either OpenSuSE ™10.3 or
Ubuntu™ "Gutsy Gibbon".

Here is a high-level diagram of our network.

As shown in this diagram, the Xen system has three physical network
interfaces. These are:

eth0 -- connected to our
DSL "Modem".

eth1 -- connected to the
switch in my office.

eth2 -- connected to a
Wireless Access Point (WAP) that interfaces to our wireless
network.

There are three Xen domains.

Dom0 (DNS name gateway.shorewall.net) is used as our main
firewall and wireless gateway as well as a local file server. It hosts
Squid running as a
transparent HTTP proxy and a DHCP server that manages IP address
assignment for both the LAN and the Wireless network.

A DomU (Domain name lists, DNS
name lists.shorewall.net) that is
used as a public Web/FTP/Mail/DNS server.

A DomU (Domain name test, DNS
name test.shorewall.net) that I use
for Shorewall testing.

Shorewall runs in Dom0.

Caution

As the developer of Shorewall, I have enough experience to be very
comfortable with Linux networking and Shorewall/iptables. I arrived at
this configuration after a fair amount of trial and error
experimentation (see Xen and the art of
Consolidation). If you are a Linux networking novice, I
recommend that you do not attempt a configuration like this one for your
first Shorewall installation. You are very likely to frustrate both
yourself and the Shorewall support team. Rather I suggest that you start
with something simple like a standalone
installation in a DomU; once you are comfortable with that then
you will be ready to try something more substantial.

Domain Configuration

Below are the relevant configuration files for the two domains. I
use a partition on my hard drives for the DomU storage device.

There is not much documentation about how to configure Xen for
routed operation. I've tried to mark the relevant parts with bold font.

Important

The files from /etc/xen/auto shown below correspond to
my configuration under Xen 3.0. I'm now running Xen 3.1 which does
not use configuration files for the domains but rather keeps the
configuration in a database managed by xend. See below.

Note that the vifname is set to 'eth3' for the virtual
interface to this DomU. This will cause the Dom0 interface to the
server to have a fixed name (eth3) which makes it a lot easier to
deal with in Shorewall and elsewhere.

Specifying an IP address (ip=206.124.146.177) causes the
vif-route script to create a host route to that IP address on
eth3.

…
# It is possible to use the network-bridge script in more complicated
# scenarios, such as having two outgoing interfaces, with two bridges, and
# two fake interfaces per guest domain. To do things like this, write
# yourself a wrapper script, and call network-bridge from it, as appropriate.
#
#(network-script network-bridge)
…
# If you are using only one bridge, the vif-bridge script will discover that,
# so there is no need to specify it explicitly.
#
#(vif-script vif-bridge)
## Use the following if network traffic is routed, as an alternative to the
# settings for bridged networking given above.
(network-script network-route)
(vif-script vif-route)

Important

As of this writing, the vif-route script does not set up
Proxy ARP correctly. So the domU can communicate with the dom0
but not with hosts beyond the dom0.

If you configure Shorewall as described below, Shorewall
will correct the Proxy ARP configuration so that it will
work.

With the three Xen domains up and running, the system looks as
shown in the following diagram.

The zones correspond to the Shorewall zones in the Dom0
configuration.

Readers who are paying attention will notice that eth4 has the
same public IP address (206.124.146.176) as eth0 (and eth3), yet the
test system connected to that interface
has an RFC 1918 address (192.168.1.7). That configuration is established
by Xen which clones the primary IP address of eth0 on all of the routed
virtual interfaces that it creates. test is configured with its default route via
192.168.1.254 which is the IP address of the firewall's br0. That works
because of the way that the Linux network stack treats local IPv4
addresses; by default, it will respond to ARP "who-has" broadcasts for
any local address and not just for the addresses on the interface that
received the broadcast (but of course the MAC address returned in the
"here-is" response is that of the interface that received the
broadcast). So when test broadcasts
"who-has 192.168.1.254", the firewall responds with "here-is
192.168.1.254 00:16:3e:83:ad:28" (00:16:3e:83:ad:28 is the MAC of
virtual interface eth4).

Caution

Under some circumstances, UDP and/or TCP communication from a
DomU won't work for no obvious reason. That happened with the
lists domain in my setup. Looking at
the IP traffic with tcpdump -nvvi eth1 in Dom0
showed that UDP packets from the lists DomU had incorrect checksums. That
problem was corrected by arranging for the following command to be
executed in the lists and test domains when the eth0 device was brought up:

ethtool -K eth0 tx off

Under OpenSuSE™ 10.2, I placed the
following in
/etc/sysconfig/network/ifcfg-eth-id-00:16:3e:b1:d7:90
(the config file for eth0):

ETHTOOL_OPTIONS='-K iface tx off'

Under other distributions, the technique will vary. For example,
under Debian™ or Ubuntu™,
you can just add a 'post-up' entry to
/etc/network/interfaces as shown here:

Caution

Update. Under OpenSuSE 10.2, communication from a domU works
okay without running ethtool but traffic shaping
in dom0 doesn't work! So it's a good idea to run it just to
be safe.

Dom0 Shorewall Configuration

In Dom0, I run a conventional three-interface firewall with Proxy
ARP DMZ -- it is very similar to the firewall described in the Shorewall Setup Guide with the
exception that I've added a fourth interface for our wireless network.
The firewall runs a routed OpenVPN
server to provide road warrior access for our three laptops and
a bridged OpenVPN server for the wireless network in our home. Here is
the firewall's view of the network:

The three laptops can be directly attached to the LAN as shown
above or they can be attached wirelessly -- their IP addresses are the
same in either case; when they are directly attached, the IP address is
assigned by the DHCP server running in Dom0 and when they are attached
wirelessly, the IP address is assigned by OpenVPN.

The Shorewall configuration files are shown below. All routing and
secondary IP addresses are handled in the OpenSuSE network
configuration.

/etc/shorewall/masq (Note the cute trick here and in
the following proxyarp file that allows me to
access the DSL "Modem" using its default IP address
(192.168.1.1)). The leading "+" is required to place the
rule before the SNAT rules generated by entries in
/etc/shorewall/nat above.