High Availability Messaging Solution

Using the AXIGEN Mail Server, Heartbeat and DRBD

July 8, 2008

Page 2 of 17 -
.
Notices
References in this publication to GECAD TECHNOLOGIES S.A. GECAD TECHNOLOGIES S.A.0 or higher of the AXIGEN Mail Server. electronic or mechanical. including photocopying and recording.com Website: http://www. All rights reserved. including photocopying and recording. No part of this document may be reproduced or transmitted in any form or by any means. If you find any problems in the documentation.A. products.. This document is copyrighted and all rights are reserved by GECAD TECHNOLOGIES S. or by any information storage and retrieval system without prior permission in writing from GECAD TECHNOLOGIES S. or services do not imply that GECAD TECHNOLOGIES S. are the user's responsibility.
Copyright Acknowledgement
(c) GECAD TECHNOLOGIES S. sales department.com Technical support: support@axigen. using: sales@axigen. Supplying this document does not give you any license to these patents.A.A.A.com. GECAD TECHNOLOGIES S.A.AXIGEN .A. You can send license inquiries. 2nd fl. product or service names may be trademarks or service marks of others.A.
. operates. GECAD TECHNOLOGIES and AXIGENTM are trademarks of GECAD TECHNOLOGIES S.A.A. Bucharest 2. in writing. except those expressly designated by GECAD TECHNOLOGIES S. to the GECAD TECHNOLOGIES S.A. will not be responsible for any loss.Increasing Service Availability Using Heartbeat & DRBD
Copyright & Trademark Notices
This article applies to version 6. programs. or by any information storage and retrieval system without the permission in writing from GECAD TECHNOLOGIES S. costs or damages incurred due to the use of this documentation. This document is copyrighted and all rights are reserved by GECAD TECHNOLOGIES S.com (c) Copyright GECAD TECHNOLOGIES S. ROMANIA Phone: +40-21-303-2080 Fax: +40-21-303-2081 Sales: sales@axigen.axigen.. Connect Business Center. 2008.A. All rights reserved. All rights reserved. electronic or mechanical. Address: 10A Dimitrie Pompei Blvd.
GECAD TECHNOLOGIES S. Evaluation and verification of operation in conjunction with other products. The information contained in this document is subject to change without notice. Other company. AXIGENTM Mail Server is a SOFTWARE PRODUCT of GECAD TECHNOLOGIES S. please report them to us in writing.A. No part of this document may be reproduced or transmitted in any form or by any means. intends to make these available in all countries in which GECAD TECHNOLOGIES S. 2008..A.A. may have patents or pending patent applications covering subject matter in this document.

having in place the proper name for IP resolution mechanisms such as DNS entries or updated/etc/hosts file.S.WebAdmin).Page 4 of 17 -
. we will assume that single points of failure are eliminated via redundant components such as multiple communication paths between nodes.2 Cluster Nodes
We will consider two identical systems where hardware components are subject to local requirements for the solution performance and resource availability.Increasing Service Availability Using Heartbeat & DRBD
1. For such a setup.
1.2 Architecture
We will focus on a two node cluster architecture in an ACTIVE/PASSIVE configuration where a designated standby host will be available to relocate our resources in case of a failure encountered in the primary system. truly innovative in several respects.cluster.cluster.x.1 Products Overview
AXIGEN Mail Server – a carrier-class messaging solution.
1. Logging.3. DRBD (Distributed Replicated Block Device) – software solution for mirroring the contents of block devices between servers. Reporting and FTP Backup modules and provides various. From now on.3.3 Hardware & Software Considerations
1.node
Note: These hosts are considered accessible via the above mentioned identification format.AXIGEN .
Note: As Heartbeat provides two distinct versions of configuration.1 O. independent power supplies for the two hosts etc.
1.
1. but you can locate precompiled packages for various Linux distributions or make use of the source code available on their respective homepages for both Heartbeat and DRBD projects. POP3. DRBD eliminates the need of using a shared storage device between our cluster nodes running AXIGEN.
. flexible administration options (including a central Web administration interface . both methods will be detailed as each of them presents a number of advantages that can suit different design schemes. This messaging solution offers the entire range of email services (SMTP. we will refer to these nodes as:
Hostname/FQDN primary. Platform
This article focuses on CentOS 5. IMAP. Heartbeat (part of the Linux-HA project) – cluster management framework for HA architectures. Prerequisites
This article is intended as a general overview on designing HA solutions using DRBD and Heartbeat along with the AXIGEN Mail Server. includes a List server. particularly scalable and configurable.node backup. WebMail). by using either v1 (legacy) or v2 (CRM mode configuration).

Page 5 of 17 -
.8.3. For a full reference on the available plugins. Additional details are provided in the Heartbeat configuration section. including here software RAID devices.8.168.node
On both nodes.
1. Gigabit Ethernet).3.8.168. multiple network adapters are required for DRBD/Heartbeat messages. The coordinates for this sample fencing device are:
IP address – 192. we will consider two similar lower level devices defined from now on as /dev/sdb1 Other types of block devices on the system can also be used as a lower level device.222 primary.node backup.
As a reference for our setup.cluster. This traffic should be managed by dedicated connections avoiding the use of actual production links. you should ensure to have designed redundant links for communication purposes.AXIGEN . If switches are to be used. you can make use of the following command:
# /usr/sbin/stonith -L apcmaster apcmastersnmp apcsmart baytech cyclades external/ibmrsa external/ibmrsa-telnet external/ipmi external/rackpdu external/riloe external/ssh external/vmware ibmhmc meatware null nw_rpc100s rcd_serial rps10 ssh suicide wti_nps
Note: The above command assumes that the corresponding heartbeat-stonith package is installed on your system.3 Block Devices
On both nodes.4 Power Fencing Devices
The STONITH module provides a number of plugins capable of managing a wide range of devices and methods for isolating cluster nodes in case of a failure.cluster.
1. As a requirement for DBRD.168. we will consider a STONITH device (APC MasterSwitch PDU) accessible via the apcmastersnmp STONITH plugin. you should consider a back-to-back approach on the communication link (For example. LVM Logical Volume etc.225 SNMP port – 161 Community – private
.Increasing Service Availability Using Heartbeat & DRBD
Example:
/etc/hosts 192.221 192.

2. For example. configuration files.Distributed Replicated Block Device
2.
. Protocol C is a synchronous replication protocol where local write operations are considered completed only after having being confirmed on both nodes. syncer { rate 10M.Increasing Service Availability Using Heartbeat & DRBD
2.AXIGEN . }
The common section contains policies that will be inherited by all defined resources.containing all user space tools.1. Installation
DRBD is available via two packages providing complementary functionalities: • DRBD kernel module . initscripts
For our CentOS platform the installation procedure can consist in using yum for this purpose:
# yum install drbd82 kmod-drbd82
Note: It may also be required to update the kernel tool too. } }
The resource section contains a collection of information that characterizes our replicated storage device. Example:
common { protocol C. DRBD .Page 6 of 17 -
. Example:
global { usage-count no. as a dependency of the DRBD kernel module.conf /etc/
A basic sample for the DRBD configuration file consists of several sections: • global • common • resource
All global parameters such as usage-count will be defined in the global section.implements the core functionalities of DRBD • DRBD package . For this reason you should copy this file in /etc/ and modify its contents according to your needs.
# cp /usr/share/doc/drbd82-*/drbd.2 Configuring DRBD
A configuration example of DRBD is provided within the documentation files and can be used as a reference for the sections and parameters supported. we will define here the replication protocol in use.

disk /dev/sdb1.168.AXIGEN . meta-disk internal.cluster.221:7788. address 192. other read/write operations on this device are not allowed. the status of this resource on each peer is also characterized by a role: Primary or Secondary.168.
Note: A split-brain condition is encountered when the cluster nodes end up with having a different set of data. } on backup. including peer authentication mechanisms. } }
In the above sample our resource has been defined as axigenStorage and all details about disk configuration. meta-disk internal. Besides these coordinates expressed in the DRBD configuration file.cluster.8. device definition. disk /dev/sdb1. peer nodes and network configuration have been provided as parameters.Page 7 of 17 -
.
Additional policies are available for definition. synchronization details and actions in case of split-brain detection.node { device /dev/drbd0.
Note: Primary role – the DRBD device is available for read/write operations.222:7788.Increasing Service Availability Using Heartbeat & DRBD
Example:
resource axigenStorage { on primary.
. address 192. The DRBD documentation provides a number of mechanisms for dealing with and preventing split-brain conditions.8. Secondary role – the DRBD device only receives updates from the peer node.node { device /dev/drbd0.

we should initialize the device metadata. our steps should comprise from enabling DRBD and then replicate the data between the two nodes. different parts of the AXIGEN storage and internal data could be placed as our axigenStorage resource defined under /etc/drbd.configuration files .--overwrite-data-of-peer primary axigenStorage
on
our
primary
node
Note: At this point.2 AXIGEN Installation
We will consider a fresh installation approach on both nodes.Increasing Service Availability Using Heartbeat & DRBD
3.cluster. we are required to ensure that the drbd module is loaded on both nodes: # modprobe drbd Next.filters/ .
.containing the main internal data of AXIGEN including: .1 AXIGEN Overview
Based on our requirements regarding data that needs to be kept synchronized between the two nodes. will be replicated on both nodes.log files .AXIGEN .conf.all user filters and also server/domain SMTP Routing and Advanced Message Rules .domains/ – hosted domains [.log/ .node): # drbdadm -. our example will consider that the entire /var/opt/axigen/ location will be our resource (axigenStorage).containing all AXIGEN binaries. attach the local device and connect our resource with the counterpart peer:
# drbdadm create-md axigenStorage # drbdadm attach axigenStorage # drbdadm syncer axigenStorage # drbdadm connect axigenStorage
The initial synchronization process consists in running (primary. As our main resource.run/ . On a Linux distribution.] If locations such as the ones for the AXIGEN binaries and intiscripts do not raise interest. /var/opt/axigen/ . the AXIGEN internal structure is formed from: • various initscripts and initscript configuration data /opt/axigen/bin/ . AXIGEN Mail Server
3.Page 8 of 17 -
.
3.. axigenStorage (made from the /var/opt/axigen/ directory).. we have promoted the axigenStorage resource to a Primary role on our main machine. as this data will only change during upgrade operations. For this process.

444 (10.
For the rest of the article. every write operation on the virtual DRBD device block found on the main cluster node will also be replicated on the backup node where the axigenStorage resource has assigned a Secondary role.ext3 /dev/drbd0 mkdir /var/opt/axigen mount /dev/drbd0 /var/opt/axigen
The installation and configuration procedures for the AXIGEN Mail Server are available under the INSTALL file provided within the package downloaded from the AXIGEN website... we can now create a filesystem on the virtual block device and mount /dev/drbd0 under /var/opt/axigen/ Example:
mkfs. Also. in our approach.AXIGEN ..node..5 (api:88/proto:86-88) GIT-hash: 9faf052fdae5ef0c61b4d03890e2d2eab550610c build by buildsvn@c5-i386-build.480) K/sec resync: used:1/31 hits:8519 misses:19 starving:0 dirty:0 changed:19 act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0
From this moment on... primary. 2008-0501 03:43:30 0: cs:SyncSource st:Secondary/Secondary ds:UpToDate/Inconsistent C r--ns:136424 nr:0 dw:0 dr:136928 al:0 bm:53 lo:0 pe:5 ua:16 ap:0 [=========>. This includes the AXIGEN binaries/initscripts/etc..cluster.Page 9 of 17 -
..1% (129952/266240)K finish: 0:00:11 speed: 11. we will consider that: • the AXIGEN service is configured and that it has been stopped and disabled from being initialized via init: Example:
# chkconfig axigen off # /etc/init.d/axigen stop
•
all drbd components are stopped
. On our main machine. the /var/opt/axigen/ location should be available as a mount point for /dev/drbd0..
Note: AXIGEN components that will not be replicated between the two nodes must be manually added on the backup server.Increasing Service Availability Using Heartbeat & DRBD
Synchronization snapshot:
# cat /proc/drbd version: 8.] sync'ed: 53.2.

in case of resource failure.2 Configuration
For the Heartbeat v1 release style.d/ # cp /usr/share/doc/heartbeat-*/authkeys /etc/ha.cf /etc/ha. sysvconfig. Heartbeat
Based on the setup requirements. resource relocation and resource collocation rules.Increasing Service Availability Using Heartbeat & DRBD
4.d) Example:
# chkconfig drbd on
4. such as monitoring operations. update-rc.d/ # cp /usr/share/doc/heartbeat-*/haresources /etc/ha. The drbdisk only includes promotion/demotion capabilities. rc-update.Page 10 of 17
. it is required that init is configured to automatically start the drbd service (Example of mechanisms for this purpose.1.d/
. depending on the OS platform: chkconfig. restart operations or relocations policies can be defined
4. we will consider the drbdisk resource agent as designated to handle our axigenStorage DRBD resource.1 Heartbeat v1 Configuration Style (Legacy Approach)
4. you may need to install the following packages: heartbeat heartbeat-stonith heartbeat-gui heartbeat-pils Example:
# yum install heartbeat heartbeat-gui
Depending on the actual architecture and resource management policies.1. as a result.1 Prerequisites
With this release of Heartbeat. three configuration files are required: # cp /usr/share/doc/heartbeat-*/ha.AXIGEN . you can make use of the following: Heartbeat v1 style configuration The Heartbeat v1 configuration style is considered a legacy approach and comes with a number of characteristics: • ease of installation and configuration • limited to two node architectures • does not provide resource health monitoring capabilities • resources configuration data is extracted via the haresources file • any changes of the cluster and configuration data must be manually replicated Heartbeat v2 (CRM style configuration) • eliminates the R1 two node architecture limit • configuration details are now part of the CIB (Cluster Information Base) XML structure • requires an understanding of this XML based configuration structure • subsequent updates are automatically replicated on all cluster nodes • resource monitoring is included and.

Page 11 of 17
. Example:
logfile /var/log/ha-log keepalive 2 deadtime 30 warntime 10 initdead 120 udpport 694 bcast eth1 eth2 auto_failback off stonith_host * apcmastersnmp 192.cf The main cluster configuration file contains details on the nodes taking part in the cluster system.8.8.168.cluster.node drbddisk::axigenStorage Filesystem::/dev/drbd0::/var/opt/axigen::ext3 192.cluster.Increasing Service Availability Using Heartbeat & DRBD
• ha. The resources handled by Heartbeat using this configuration style will be defined as a group..authkeys • authkeys The file structure is composed of an authentication mechanism expressed in the format of:
auth method-id method-id method key
Example:
auth 1 1 sha1 secret
Given the nature of the data contained here.node # Log file # # Counters and # Threshold values # # UDP port # communication details # resource transition policy # example stonith device # cluster nodes
Authentication File . one per line:
cluster-node resource1 resource2 .226 axigen
.AXIGEN .168..cluster.node backup. communication paths (serial/Ethernet adapters) and methods (unicast/broadcast/multicast) based on the network architecture details. you should ensure that this file is not world readable: # chmod 600 /etc/ha.225 161 private node primary.preferred system for running the goup of resources
Note: The resources of such a group will be started from the left to the right and stopped from the right to the left.
Example:
primary.d/authkeys Heartbeat v1 configuration makes use of the haresources file for resource definition. Multiple counters and thresholds are also available for configuration in this configuration file. resourceN cluster-node .

in our case we have assumed an ext3 filesystem • start our virtual IP address resource • startup AXIGEN via available intiscript
Note: As the drbdisk only implements promotion/demotion facilities.node peer via drbdisk • mount /dev/drbd0 under /var/opt/axigen/. or the hb_gui interface) should be used to generate and manage the cluster configuration details. rc-update.py
. For this process. sysvconfig.Page 12 of 17
. as detailed in section I.
To facilitate the migration from the legacy mode to this new structure. Heartbeat also provides a tool for conversion to the CIB structure: # /usr/lib/heartbeat/haresources2cib.
/etc/init. external provisioning tools must be deployed. usually located under: /var/lib/heartbeat/crm/cib.d/drbd restart
By starting Heartbeat on both hosts.
Heartbeat should also be initialized by init during system startup. XML editors and Heartbeat administration tools (such as cibadmin. update-rc. Example:
crm yes
The haresources file is no longer necessary as the v2 release makes use of a XML configuration file.cluster. one needs to ensure that the drbd service is running on both nodes.cf file we need to append a corresponding entry stating that we will in fact make use of the v2 release style configuration. In the ha. you can make use of a number of tools: chkconfig.d Example:
# chkconfig heartbeat on
4.
Example:
# /etc/init.2 Heartbeat v2 Configuration Style (CRM Mode)
Configuring Heartbeat v2. all our resources will be started on the primary cluster node and our designated backup system will be ready to relocate the resources in the event of a failure on our main node.AXIGEN .xml
Note: This file should not be manually edited.Increasing Service Availability Using Heartbeat & DRBD
The order for starting the resources: • promoting the axigenStorage resource to a primary role on the primary. Instead.2 supports some modifications as to reflect the CRM style method.d/heartbeat start
Note: For resource health monitoring operations.

enabling STONITH devices and configuring the actions that should be triggered (reboot/poweroff). the cluster does not monitor the resource health.1 Cluster Wide Configuration Attributes
Example:
<cib admin_epoch="0" epoch="1" num_updates="1"> <configuration> <crm_config> <cluster_property_set id="sample-options"> <attributes> <nvpair id="require_quorum" name="require_quorum" value="true"/> <nvpair id="symmetric_cluster" name="symetric_cluster" value="true"/> <nvpair id="suppress_cib_writes" name="suppress_cib_writes" value="true"/> <nvpair id="stonith_enabled" name="stonith_enabled" value="true"/> <nvpair id="stonith_action" name="stonith_action" value="reboot"/> [. The tuple formed via (admin_epoch.containing information about cluster nodes.2.2. this option being available by creating specific monitoring operations:
.Page 13 of 17
. resources. epoch. location policies • status ..maintained by the cluster software itself The configuration section requires managing operations from the administrator side and is composed from four parts:
<configuration> <crm_config/> <nodes/> <resources/> <constraints/> </configuration>
4.2 Resources
We will define a group of resources that make up all the elements required for running AXIGEN: • virtual interface containing details such as IP address/broadcast address/netmask value • device hosting our /var/opt/axigen/ data including information on the filesystem itself • AXIGEN initscript By default. The node with the highest tuple will replace the configuration details on the rest of the peers.AXIGEN . num_updates) provides details on the best configuration to be used by the system.] </attributes> </cluster_property_set> </crm_config>
We can configure attributes that control the overall behavior of our cluster system such as symmetric_cluster allowing resources to run on any node of the cluster.
4.Increasing Service Availability Using Heartbeat & DRBD
The cluster configuration structure is divided into two main sections: • configuration ..