Menu

Setup of Ganeti cluster on root servers of Hetzner

01 November 2014

I'll setup a 3 node Ganeti cluster on Hetzners root servers, to run static and Drupal based sites for friends and NGOs. Ganeti is designed to facilitate cluster management of virtual servers (the "instances") using commodity hardware (the "nodes"). It provides fast and simple recovery after physical failures, disk creation management, operating system installation, startup, shutdown, and failover between physical systems. Ganeti is built on top of existing virtualization technologies such as Xen or KVM and other open source software. It's easy to start with one physical node, and one cluster can scale up to 150 physical nodes. Below I'll follow Ganeti terminology, and the root servers will be called "nodes". The nodes will be the KVM hosts for guest OS systems. These guest systems (or the virtual machines) are named to "instances".

The recent Ganeti stack on Debian Wheezy, as of October 2014, consists of:

Debian Wheezy 7.7

DRBD 8.3.11

KVM 1.1.2

Ganeti 2.11.5.

This will definitely change over time, of course. Next to the Ganeti stack, some other tools will be also installed: pound as reverse-proxy and load balancer, Shorewall and fail2ban as firewall to reject malicious IP addresses.

Get root servers with Debian image

First order root servers with one additional IP. Register at Hetzner, and order, either some new E40 servers, or go for server bidding to bargain. After 30-60 minutes and some confirmation emails containing the node IP address and a temporary root password, you can login to your fresh root server. Mine has Intel i7-3770, 32 GB and 2x3TB disks, with traffic 10 TB per month.

Login and, in order to deploy their prepared Debian OS image on the node, run the installimage script.

Select Debian, and the 64 bit minimal image named debian-77-wheezy-64-minimal. In the editor use software RAID, and set the hostname as FQDN like this: node1.example.com

For the partition layout, set 1GB /boot directory and 3 volume groups. I take 120GB for the system volume group, including 16GB swap and the 100+GB root filesystem with ext4 for the OS, a 1500GB volume group for Ganeti, and the rest of disk space named as unused.

PART /boot ext4 1G
PART lvm system 120G
PART lvm ganeti 1500G
PART lvm unused all
LV system swap swap swap 16G
LV system root / ext4 all

Save (F2) and exit (F10), then the install process will run.

When it's ready, reboot and login again with temporary password. Check few things:

Reading all physical volumes. This may take a while...
Found volume group "unused" using metadata type lvm2
Found volume group "ganeti" using metadata type lvm2
Found volume group "system" using metadata type lvm2

Networking

via icfonfig,

and so on...

Basic Debian config

Reboot and login again with temporary password, and start with locales.

locales

The locales should be reconfigured first, before changing passwords. I select my local languages, as the second language, next to English, and set English for system language. Defaults will be en_US.UTF8 and hu_HU.UTF8 for me. Use either:

reboot

Now login to the new server as root, it should happen without asking for a password, and you can restart the SSH service or make a reboot:

/etc/init.d/ssh restart

Setup networking for Ganeti

So we have now an up-to-date Debian Wheezy server with a bit hardened SSH. Let’s continue with networks, IP addresses, hostname, DNS and setting up the networking mode for Ganeti.

Each node will have an IP for the host access, and an additional public IP, used for cluster IP in case of master node, otherwise unused. The loadbalancer ans firewall will run on node, everything else on virtual machines (instances). And Hetzner has also some rules regarding their network architecture.

Config checks

Check few relevant setting, like hostname and DNS resolution. Small but important, always use FQDN in /etc/hostname. Default should be OK, but check it:

cat /etc/hostname

DNS resolution check:

dig google.com

Setting in /etc/hosts

Manually define the DNS settings for all the nodes and instances by updating the /etc/hosts file.

Install and config DRBD

DRBD is RAID1 over the network, and is required for high availability. The stable (default) Debian Wheezy repository contains DRBD 8.3.13 and Ganeti 2.5.2, so we have to use the backport repository to get Ganeti 2.11.5.

Add the the Debian backports repo to the source list /etc/apt/sources.list

Setup the OS templating tool

In Ganeti, unlike virsh, you can't just install any OS from ISO image.
There are some options:
* ganeti-instance-debootstrap: for Debian and Ubuntu (default, included in ganeti2 package)
* ganeti-instance-image (from ganeti-instance-image apt repository)
raw

We will use ganeti-instance-image, since it simply works and any ISO image can be used to create instances. Ganeti Instance Image is the guest OS definition for Ganeti that uses either file system dumps or tar ball images to deploy instances . See howto: http://notes.ceondo.com/ganeti/

Install Ganeti Instance Image

First add the ganeti-instance-image repo to the source list in /etc/apt/sources.list