Introduction

In this howto we will build a load-balanced and high-availability web cluster on 2 real servers with Xen, hearbeat and ldirectord. The cluster will do http, mail, DNS, MySQL database and will be completely monitored. This is currently used on a production server with a couple of websites.

The goal of this tutorial is to achieve load balancing & high availability with as few real servers as possible and of course, with open-source software. More servers means more hardware & hosting cost.

Most of the information you will find here has been copy / pasted from a dozen howtos, many of them from howtoforge.com, but some important details have been modified to make this possible and to put everything together.

Here is a quick list of services & applications that will be installed:

Apache

MySQL + phpmyadmin

Postfix (SMTP) with web based users configuration and Spamassassin

Courier (IMAP & POP) and squirrelmail

Bind (DNS server)

Munin and monit for web based monitoring

Homemade scripts for monitoring

What you need

2 servers with dual lan, at least 7 IPs. IPs will be used like this :

dom01.example.com : 192.168.1.100

dom02.example.com : 192.168.1.101

lb1.example.com : 192.168.1.102

lb2.example.com : 192.168.1.103

web1.example.com : 192.168.1.104

web2.example.com : 192.168.1.105

example.com : 192.168.1.106

yousite.com (optional) : 192.168.1.107

Dom0 will be separated from load balancers and web servers. I didn't try it but I believe it would be possible to put load balancers on Dom0.

I suggest at least 2GB ram and RAID 1 or 10 hard drives for a production server.

Limitations

1) This worked for me. Doesn't mean it will work for you but rest assured that the howto is 100% tested to work on a production and test server !

2) This setup is scalable over 2 servers but you will need to find another way for MySQL replication if you do so.

3) No control panel such as ISPConfig, CPanel, etc...

4) Some websites can break MySQL Master to Master replication. It happend to me with Drupal but I fixed it either by disabling cache or by setting a minimum cache lifetime. Please read this before you go further :

A: MySQL replication currently does not support any locking protocol between master and slave to guarantee the atomicity of a distributed (cross-server) update. In other words, it is possible for client A to make an update to co-master 1, and in the meantime, before it propagates to co-master 2, client B could make an update to co-master 2 that makes the update of client A work differently than it did on co-master 1. Thus, when the update of client A makes it to co-master 2, it produces tables that are different from what you have on co-master 1, even after all the updates from co-master 2 have also propagated. This means that you should not chain two servers together in a two-way replication relationship unless you are sure that your updates can safely happen in any order, or unless you take care of mis-ordered updates somehow in the client code.

1. Installing Ubuntu

Do a basic install of Ubuntu 8.04 LTS server edition.

If you want to install with software RAID 1 please read this howto I wrote :

You need to do 2 Xen domain on each server (dom01 and dom02 are Dom0 or VM controller) :

server #1 - dom01.example.com

lb1.example.com (256MB RAM - 5GB HD is enough)
ip : 192.168.1.102

web1.example.com (the more RAM the better, keep 512MB for Dom0)
ip : 192.168.1.104

server #2 - dom02.example.com

lb2.example.com (256MB RAM - 5GB HD is enough)
ip : 192.168.1.103

web2.example.com (the more RAM the better, keep 512MB for Dom0)
ip : 192.168.1.105

3. Creating Xen Bridges for local data transfers (optional)

By default only one network card is enabled on virtual machine with Xen. For local transfer such as rsync, MySQL replication and backups I use a gigabit crossover cable between the 2 servers. Its not necessary but it will savebandwidth costs and replication will be faster.

4.1 Installing openssh server and VIM

4.2 Updating the repositories

#
# deb cdrom:[Ubuntu-Server 8.04 _Hardy Heron_ - Release i386 (20080423.2)]/ hardy main restricted
#deb cdrom:[Ubuntu-Server 8.04 _Hardy Heron_ - Release i386 (20080423.2)]/ hardy main restricted
# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution.
deb http://de.archive.ubuntu.com/ubuntu/ hardy main restricted
deb-src http://de.archive.ubuntu.com/ubuntu/ hardy main restricted
## Major bug fix updates produced after the final release of the
## distribution.
deb http://de.archive.ubuntu.com/ubuntu/ hardy-updates main restricted
deb-src http://de.archive.ubuntu.com/ubuntu/ hardy-updates main restricted
## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team, and may not be under a free licence. Please satisfy yourself as to
## your rights to use the software. Also, please note that software in
## universe WILL NOT receive any review or updates from the Ubuntu security
## team.
deb http://de.archive.ubuntu.com/ubuntu/ hardy universe
deb-src http://de.archive.ubuntu.com/ubuntu/ hardy universe
deb http://de.archive.ubuntu.com/ubuntu/ hardy-updates universe
deb-src http://de.archive.ubuntu.com/ubuntu/ hardy-updates universe
## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team, and may not be under a free licence. Please satisfy yourself as to
## your rights to use the software. Also, please note that software in
## multiverse WILL NOT receive any review or updates from the Ubuntu
## security team.
deb http://de.archive.ubuntu.com/ubuntu/ hardy multiverse
deb-src http://de.archive.ubuntu.com/ubuntu/ hardy multiverse
deb http://de.archive.ubuntu.com/ubuntu/ hardy-updates multiverse
deb-src http://de.archive.ubuntu.com/ubuntu/ hardy-updates multiverse
## Uncomment the following two lines to add software from the 'backports'
## repository.
## N.B. software from this repository may not have been tested as
## extensively as that contained in the main release, although it includes
## newer versions of some applications which may provide useful features.
## Also, please note that software in backports WILL NOT receive any review
## or updates from the Ubuntu security team.
# deb http://de.archive.ubuntu.com/ubuntu/ hardy-backports main restricted universe multiverse
# deb-src http://de.archive.ubuntu.com/ubuntu/ hardy-backports main restricted universe multiverse
## Uncomment the following two lines to add software from Canonical's
## 'partner' repository. This software is not part of Ubuntu, but is
## offered by Canonical and the respective vendors as a service to Ubuntu
## users.
# deb http://archive.canonical.com/ubuntu hardy partner
# deb-src http://archive.canonical.com/ubuntu hardy partner
deb http://security.ubuntu.com/ubuntu hardy-security main restricted
deb-src http://security.ubuntu.com/ubuntu hardy-security main restricted
deb http://security.ubuntu.com/ubuntu hardy-security universe
deb-src http://security.ubuntu.com/ubuntu hardy-security universe
deb http://security.ubuntu.com/ubuntu hardy-security multiverse
deb-src http://security.ubuntu.com/ubuntu hardy-security multiverse

Now do :

apt-get update
apt-get upgrade

4.3 Modifications

/bin/sh is a symlink to /bin/dash, however we need /bin/bash, not /bin/dash. Therefore we do this:

ln -sf /bin/bash /bin/sh

We will disable AppArmor (on dom01 and dom02) by doing the following :

/etc/init.d/apparmor stop
update-rc.d -f apparmor remove

5. Network configuration (dom01, dom02, lb1, lb2, web1, web2)

5.1 Setting up IPs

To edit network configuration under Ubuntu do :

vi /etc/network/interfaces

We will now do each network configuration one by one. I assume you use 2 network card, eth0 is the one connected to the internet and eth1 the one with the crossover cable. I wont write the config file individually, only for dom01.example.com, please modify accordingly to this list :