Xen and the Art of Consolidation

TomEastep

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version
1.2 or any later version published by the Free Software Foundation; with
no Invariant Sections, with no Front-Cover, and with no Back-Cover
Texts. A copy of the license is included in the section entitled
“GNU Free Documentation
License”.

Caution

This article applies to Shorewall 3.0 and later. If you are running
a version of Shorewall earlier than Shorewall 3.0.0 then please see the
documentation for that release.

Xen Network Environment

Xen is a
paravirtualization tool that allows you to run
multiple virtual machines on one physical machine. It is available on a
wide number of platforms and is included in recent
SUSE™ distributions.

Xen refers to the virtual machines as
Domains. Domains are numbered with the first domain
being domain 0, the second domain 1, and so on. Domain 0
(Dom0) is special because that is the domain
created when the machine is booted. Additional domains (called
DomU's) are created using the xm
create command from within Domain 0. Additional domains can also
be created automatically at boot time by using the
xendomains service.

Xen virtualizes a network interface named eth0[1]in each domain. In Dom0, Xen also creates a bridge (xenbr0) and a number of virtual interfaces
as shown in the following diagram.

I use the term Extended Dom0 to distinguish
the bridge and virtual interfaces from Dom0 itself. That distinction is
important when we try to apply Shorewall in this environment.

The bridge has a number of ports:

peth0 — This is the port that connects to the physical network
interface in your system.

vif0.0 — This is the bridge port that is used by traffic to/from
Domain 0.

vifX.0 — This is the bridge port that is used by traffic to/from
Domain X.

Before Xen

Prior to adopting Xen, I had a home office crowded with 5 systems,
three monitors a scanner and a printer. The systems were:

Firewall

Public Server in a DMZ (mail)

Private Server (wookie)

My personal Linux Desktop (ursa)

My work system (docked laptop running Windows XP).

The result was a very crowded and noisy room.

After Xen

Xen has allowed me to reduce the noise and clutter considerably. I
now have three systems with two monitors. I've also replaced the
individual printer and scanner with a Multifunction
FAX/Scanner/Printer.

My Linux desktop (wookie, which is actually the old public
server box)

Most of the Linux systems run SUSE ™10.1; my
personal Linux desktop system and our Linux Laptop run
Ubuntu™ "Dapper Drake".

The configuration described below uses a
bridged Xen Networking configuration; if you want to see how to accomplish
a similar configuration using a Routed Xen configuration then please see
this article. I am now using the
routed configuration because it results in one fewer domains to
administer.

Here is a high-level diagram of our network.

As shown in this diagram, the Xen system has three physical network
interfaces. These are:

eth0 -- connected to the
switch in my office. That switch is cabled to a second switch in my
wife's office where my wife has her desktop and networked printer (I
sure wish that there had been wireless back when I strung that CAT-5
cable halfway across the house).

eth1 -- connected to our
DSL "Modem".

eth2 -- connected to a
Wireless Access Point (WAP) that interfaces to our wireless
network.

There are three Xen domains.

Dom0 (DNS name ursa.shorewall.net) is used as a local file
server (NFS and Samba).

The first DomU (Dom name firewall, DNS name gateway.shorewall.net) is
used as our main firewall and wireless gateway.

The second DomU (Dom name lists, DNS name lists.shorewall.net) is used as
a public Web/FTP/Mail/DNS server.

Shorewall runs in Dom0 and in the firewall domain.

Caution

As the developer of Shorewall, I have enough experience to be very
comfortable with Linux networking and Shorewall/iptables. I arrived at
this configuration after a fair amount of trial and error
experimentation. If you are a Linux networking novice, I recommend that
you do not attempt a configuration like this one for your first
Shorewall installation. You are very likely to frustrate both yourself
and the Shorewall support team. Rather I suggest that you start with
something simple like a standalone
installation in a domU; once you are comfortable with that then
you will be ready to try something more substantial.

With all three Xen domains up and running, the system looks as
shown in the following diagram.

The zones correspond to the Shorewall zones in the firewall DomU
configuration.

Note

If you want to run a simple NAT gateway in a Xen DomU, just omit
the second bridge (xenbr1), the second delegated interface, and the
second DomU from the above configuration. You can then install the
normal Shorewall two-interface sample
configuration in the DomU.

Caution

Under some circumstances, UDP and/or TCP communication from a
domU won't work for no obvious reason. That happened with the
lists domain in my setup. Looking at
the IP traffic with tcpdump -nvvi eth1 in the
firewall domU showed that UDP packets
from the lists domU had incorrect
checksums. That problem was corrected by arranging for the following
command to be executed in the lists
domain when its eth0 device
was brought up:

ethtool -K eth0 tx off

Under SUSE 10.1, I placed the following in
/etc/sysconfig/network/if-up.d/resettx (that file
is executable):

Dom0 Configuration

The goals for the Shorewall configuration in Dom0 are as
follows:

Allow traffic to flow unrestricted through the two bridges.
This is done by configuring the hosts connected to each bridge as a
separate zone and relying on Shorewall's implicit intra-zone ACCEPT
policy to permit traffic through the bridge.

Ensure that there is no stray traffic between the zones. This
is a "belt+suspenders" measure since there should be no routing
between the bridges (because they don't have IP addresses).

Firewall DomU Configuration

In the firewall DomU, I run a conventional three-interface
firewall with Proxy ARP DMZ -- it is very similar to the firewall
described in the Shorewall Setup
Guide with the exception that I've added a fourth interface for
our wireless network. The firewall runs a routed OpenVPN server to provide road warrior access
for our two laptops and a bridged OpenVPN server for the wireless
network in our home. Here is the firewall's view of the network:

The two laptops can be directly attached to the LAN as shown above
or they can be attached wirelessly -- their IP addresses are the same in
either case; when they are directly attached, the IP address is assigned
by the DHCP server running in Dom0 and when they are attached
wirelessly, the IP address is assigned by OpenVPN.

The Shorewall configuration files are shown below. All routing and
secondary IP addresses are handled in the SUSE network
configuration.

/etc/shorewall/masq (Note the cute trick here and in
the following proxyarp file that allows me to
access the DSL "Modem" using its default IP address
(192.168.1.1)). The leading "+" is required to place the
rule before the SNAT rules generated by entries in
/etc/shorewall/nat above.