Introduction

This is a step by step HowTo for setting up clustered Samba with CTDB on top of GFS2.

Warning: Out of date!!!

Please note that this guide is very old and doesn't contain the latest instructions for configuration CTDB and Samba. For the parts covering CTDB and Samba specific configuration please follow the various sections under CTDB and Clustered Samba.

All of the cluster nodes over which you plan to use GFS2 must be able to mount the same shared storage (Fibre Channel, iSCSI, etc.).

Fence device: Any installation must use some sort of fencing mechanism. GFS2 requires fencing to prevent data corruption. http://sources.redhat.com/cluster/wiki/FAQ/Fencing has more info about fencing. A Fence Device is not required for test clusters.

/* This list of required and optional packages for clustered samba is not complete. Please help me in filling this in. I usually configure a yum repo and do a 'yum -y install cman lvm2-cluster gfs2-utils' for my cluster and it pulls everything in automatically. */

Optional Packages

Depending on what method you use to access shared storage, additional packages might be needed. For example, you'll need aoetools if you plan to use ATAoE.

It is recommended that you use the yum package management tool to update/install these packages so you have all the necessary dependencies resolved automatically.
Something like, "yum install cman lvm2-cluster gfs2-utils ctdb samba"

Configuration

All configuration steps in this section are to be performed on all of the cluster nodes unless otherwise specified. Setting up one machine and copying the configs over to the other nodes is one way to do this if you're not using a cluster shell (like cssh or mrxvt) that can broadcast keystrokes to all the tabs/windows, each of which is connected to a different cluster node.

Cluster configuration

To set up a cluster, all you need is a cluster configuration xml file located at /etc/cluster/cluster.conf. A simple configuration for a 3 node cluster is as shown below.
Example:

The <cluster name> attribute is your name for the cluster. This needs to be unique among all the clusters you may have on the same network. As we will see later when configuring GFS2, this cluster name will also associate the filesystem with it.
For each <clusternode>, you need to specify a "name" which is also the hostname of the corresponding machine and the "nodeid", which is numeric and unique to the nodes.
Note: For a two-node cluster, there's a special attribute that needs to be part of the cluster.conf. Add <cman two_node="1" expected_votes="1"/> within the <cluster></cluster> tags.
This example below adds a power fencing (apc) device to the configuration:

Each <clusternode> has an associated <fence> attribute and a separate <fencedevices> tag within the root <cluster> tag that defines the fencing agent. http://sources.redhat.com/cluster/wiki/FAQ/Fencing has more information on configuring different fencing agents.
The cluster.conf manual page has more information on the various options available.

lvm2-cluster Configuration

Before using lvm2-cluster(clvmd) to manage your shared storage, make sure you edit the lvm configuration file at /etc/lvm/lvm.conf to change the locking_type attribute to "locking_type=3". Type 3 uses built-in clustered locking.

GFS2 Configuration

When using clvm, you must bring up the cluster first (so clvmd is running) before you can create logical volumes and gfs2 filesystems. Please skip this step and move on to the next step CTDB Configuration. We will come back to this step later.

Use the lvm tools like pvcreate, vgcreate, lvcreate to create two logical volumes ctdb_lv and csmb_lv on your shared storage. ctdb_lv will store the shared ctdb state and needs to be 1GB in size. csmb_lv will hold the user data that will be exported via a samba share so size it accordingly. Note that creation of clustered volume groups and logical volumes is to be done on only one of the cluster nodes. After creation, a "service clvmd restart" on all the nodes will refresh this new information and all the nodes will be able to see the logical volumes you just created.

You will be configuring gfs2 filesystems on both these volumes.
The only thing you need to do for gfs2 is to run mkfs.gfs2 to make the filesystem. Note:mkfs.gfs2 is to be run on one cluster node only!

In this example, we've used a 100GB (/dev/csmb_vg/csmb_lv) gfs2 filesystem for the samba share and the 1GB (/dev/csmb_vg/ctdb_lv) gfs2 filesystem for ctdb state.

First, we will create the filesystem to host the samba share.
</p>

mkfs.gfs2 -j3 -p lock_dlm -t csmb:gfs2 /dev/csmb_vg/csmb_lv

-j : Specifies the number of journals to create in the filesystem. One journal per node, and we have 3 nodes.

Not that there should be no spaces around the equals signs, since this config file is sourced by a shell script...

CTDB_NODES specifies the location of the file which contains the list of ip addresses of the cluster nodes.

CTDB_PUBLIC_ADDRESSES specifies the location of the file where a list of ip addresses can be used to export the samba shares exported by this cluster.

CTDB_RECOVERY_LOCK specifies a lock file that ctdb uses internally for recovery and this file must reside on shared storage such that all the cluster nodes have access to it. In this example, we've used the gfs2 filesystem that will be mounted at /mnt/ctdb on all nodes. This is different from the gfs2 filesystem that will host the samba share that we plan to export. This reclock file is used to prevent split-brain scenarios. With newer versions of CTDB (>= 1.0.112), specifying this file is optional, but it is strongly recommended that it is substituted with another split-brain prevention mechanism.

CTDB_MANAGES_SAMBA=yes. Enabling this allows ctdb to start and stop the samba service as it deems necessary to provide service migration/failover etc.

CTDB_MANAGES_WINBIND=yes. If running on a member server, you will need to set this too.

The contents of the /etc/ctdb/nodes file:

192.168.1.151
192.168.1.152
192.168.1.153

This simply lists the cluster nodes' ip addresses. In this example, we assume that there is only one interface/IP on each node that is used for both cluster/ctdb communication and serving clients. If you have two interfaces on each node and wish to dedicate one set of interfaces for cluster/ctdb communication, use those ip addresses here and make sure the hostnames/ip addresses used in the cluster.conf file are the same.

It is critical that this file is identical on all nodes because the ordering is important and ctdb will fail if it finds different information on different nodes.

The contents of the /etc/ctdb/public_addresses file:

192.168.1.201/24 eth0
192.168.1.202/24 eth0
192.168.1.203/24 eth0

We're using three addresses in our example above which are currently unused on the network. Please choose addresses that can be accessed by the intended clients.

These are the IP addresses that you should configure in DNS for the name of the clustered samba server and are the addresses that CIFS clients will connect to. By using different public_addresses files on different nodes it is possible to partition the cluster into subsets of nodes

We export a share with name "csmb" located at /mnt/gfs2/share. Recall that this is different from the GFS2 shared filesystem that we used earlier for the ctdb lock file at /mnt/ctdb/.ctdb.lock. We will create the "share" directory in /mnt/gfs2 when we mount it for the first time. clustering = yes instructs samba to use CTDB. netbios name = csmb-server explicitly sets all the nodes to have a common NetBIOS name.

smb.conf should be identical on all the cluster nodes.
</ul>
</li>

Bringing up the cluster, clvmd (optional), gfs2, ctdb and samba

Bringing up the Cluster

The cman init script is the best and recommended way of bringing up the cluster.
"service cman start" on all the nodes will bring up the various components of the cluster.
Note for Fedora installs: NetworkManager is enabled and run by default. This must be disabled in order to run cman.
# service NetworkManager stop# chkconfig NetworkManager offYou might have to configure network interfaces manually using system-config-network. Also see man ifconfig for more information.

The output should be similar on all the nodes. The status field "Sts" value of 'M' denotes that the particular node is a member of the cluster.

You can have a cluster node automatically join the cluster on reboot by having init start it up.

"chkconfig cman on"

Bringing up clvmd

"service clvmd start" on all the nodes should start the clvmd daemon and activate any clustered logical volumes that you might have. Since this is the first time you're bringing up this cluster, you still need to create the logical volumes required for its operation. At this point, you can go back to the previous skipped section GFS2 Configuration to create logical volumes and gfs2 filesystems as described there.

You can have a cluster node automatically start clvmd at system boot time by having init start it up. Make sure you have "cman" configured to start on boot as well.

"chkconfig clvmd on"

Bringing up GFS2

"mount -t gfs2 /dev/csmb_vg/csmb_lv /mnt/gfs2" and "mount -t gfs2 /dev/csmb_vg/ctdb_lv /mnt/ctdb" on all nodes will mount the gfs2 filesystems we created on /dev/csmb_vg/csmb_lv and /dev/csmb_vg/ctdb_lv at the mount points /mnt/gfs2 and /mnt/ctdb respectively. Recall that the "/mnt/gfs2" mountpoint was configured in /etc/samba/smb.conf and "/mnt/ctdb" was configured in /etc/sysconfig/ctbd
Once mounted, from a single node, create the share directory /mnt/gfs2/share, set it's permissions according to your requirements and copy on all the data that you wish to export. Since gfs2 is a cluster filesystem, you'll notice that all the other nodes in the cluster also have the same contents in the /mnt/gfs2/ directory.
It is possible to make gfs2 mount automatically during boot using "chkconfig gfs2 on". Make sure you have "cman" and "clvmd" also set to start up on boot.
Like all other filesystems that can be automounted, you will need to add these lines to /etc/fstab before gfs2 can mount on its own: