Creating a HA cluster using two Centos 6.x servers

Important info in regards to CentOS 6.5

Due to consequence of multilib support and PATH handling heartbeat’s init scripts wont work on Centos 6.5. The easiest way to fix it is to install heartbeat 3.0.4-2 that is not currently in the epel repo, but can be downloaded from here:

Note: you will need both heartbeat-3.0.4-2.el6.x86_64.rpm and heartbeat-libs-3.0.4-2.el6.x86_64.rpm packages. In regards to further development of this but you can check this bugzilla bug.

Prerequisites

Two ‘base’ CentOS servers (architecture is irrelevant)

A lot of time

Notes about this guide

I will presume that both CentOS servers were installed following this guide, with a partitioning scheme similar to this.

Unless otherwise specified, with redfor primary and blue for secondary, run each command on both servers.

Before we begin make sure that both servers are up to date:

yum -y update

Installing packages

We need to add a repo first, import the public key:

rpm --import http://elrepo.org/RPM-GPG-KEY-elrepo.org

and than add the repo:

rpm -Uvh http://elrepo.org/elrepo-release-6-5.el6.elrepo.noarch.rpm

now we install drbd, it’s kernel module and heartbeat, by running:

yum -y install drbd84-utils kmod-drbd84 heartbeat

Note: As of writing this document the latest drbd version is 8.4, your milage might differ, change the above command accordingly.

Configuring DRBD

DRBD refers to block devices designed as a building block to form high availability (HA) clusters. This is done by mirroring a whole block device via an assigned network. DRBD can be understood as network based raid-1.

Presuming, that you followed the custom partitioning scheme how-to you should have some free space in your LVM that you can allocate to be used by drbd. Let us create a logical volume, named lv_drbd by running:

lvcreate -L 35GB --name lv_drbd vg_linuxserver

where 35GB is the size of the block device, and vg_linuxserver is your volume group name; usually it’s the same as your host name preceded by a vg_ prefix. You can check your host name by typing:

This will create a repdata resource shared between linuxserver1 and linuxserver2. The servers IP addresses are 192.168.10.2 and 192.168.10.3 respectively. Adjust those settings to match your setup. Don’t forget to change somepassword to something more secure.

Initialize the meta-data area by running:

drbdadm create-md repdata

and start drbd:

service drbd start

check it’s status by running:

cat /proc/drbd

or:

service drbd status

As you can see , both nodes are secondary, which is normal. we need to decide which node will act as a primary now, that will initiate the first ‘full sync’ between the two nodes. Run this ONLY on the primary node:

drbdadm -- --overwrite-data-of-peer primary repdata

You can monitor the progress (on either node) of the sync by running:

watch -n 1 cat /proc/drbd

When all is done format and mount it. Run this ONLY on the primary node:

but don’t forget to create the mount point on the secondary node as well. Run this ONLY on the secondary node:

mkdir&nbsp;/repdata

Configuring Heartbeat

Heartbeat is a daemon that provides cluster infrastructure (communication and membership) services to its clients. This allows clients to know about the presence (or disappearance!) of peer processes on other machines and to easily exchange messages with them.

Pay close attention the the node lines, they should be the hostnames of the two linux servers. If you want to run more than one HA cluster inside the same network change the udpport to a different, unused one.

This basically means that linuxserver1 is the primary node, the node ip address will be 192.168.10.3 on a 255.255.255.0 subnet. The filesystem is a drbddisk, namely /dev/drdb0, that should be mounted on the /repdata mountpoint as ext4. The service that should be started is httpd (apache). Adjust it to fir your needs.

And finally start the service:

service heartbeat start

IMPORTANT NOTES

make sure that the ports listed above are open on both servers

make sure that drbd and heartbeat start at boot time

make sure to NOT start the service at boot time, since it is managed by heartbeat

make sure that the service in question actually uses the /repdata mount point.