Server Failure Recovery

Recovering from a Failed Primary Database Server

If the current primary database server and its integrated management module (IMM) lose power or otherwise fail, the current secondary server cannot automatically take over the primary role. Under these conditions, all calls to or from the system fail, and meetings cannot be scheduled or modified.

Disabling High Availability to Enable the Current Secondary Database Server to Take Over the Primary Role

The high availability (HA) implementation requires access to the IMM of the failed node to ensure that the node is no longer accessing the Distributed Replicated Block Device (DRBD) disk, which is a shared resource, before allowing a role transfer. Therefore, if the IMM interface of the current primary database server becomes unavailable, you need to complete the following procedure to manually enable the current secondary database server to take over the primary role.

Procedure

Step 1 Log in to the CLI of the database server that is still working.

Step 2 Enter the utils service database status command to verify that the node has not already taken over the primary HA role.

Note If the current HA role is primary, do not complete the rest of this procedure. You already have a working current primary database server. If the failed server needs to be replaced, proceed to the "Replacing a Database Server" section.

Step 3 Enter utils service database drbd disable-ha.

admin: utils service database drbd disable-ha

Stopping Heartbeat...

Disabling STONITH...

[Done]

Step 4 Enter the utils service database status command to verify that the node takes over the primary HA role.

Caution This procedure will temporarily interrupt MySQL service. Cisco recommends that you complete this task during a maintenance window. During the MySQL service interruption, new calls will not be able to connect to meetings, and users will not be able to schedule meetings.

Procedure

Step 1 Turn off the recovered server.

Step 2 Log in to the CLI of the current primary database server.

Step 3 Enter utils service database drbd enable-ha.

admin: utils service database drbd enable-ha

Stopping Heartbeat...

Stopping Mon...

Stopping mon daemon: [ OK ]

Stopping MySQL...

Shutting down MySQL. SUCCESS!

Unmounting DRBD Volume...

Entering DRBD Secondary mode...

Enabling STONITH...

Starting Heartbeat...

[Done]

Step 4 Turn on the recovered server but do not take any further actions on that server.

After the IMM becomes available, the HA implementation will automatically set up the peer communications and reboot the recovered node.

Preparing to Replace a Database Server

Step 1 Obtain the Cisco TelePresence Exchange System installation DVD, or download the software from the following URL and burn the disk image onto a DVD: http://www.cisco.com/go/ctx-download.

Note Make sure that the software version on the installation DVD is the same as the version that is currently running on the peer server of the same role. If you want to upgrade the software, you may do so after you successfully replace the failed server.

•Hostname, virtual IP (VIP) address, and subnet mask that are shared by both database servers.

•Default gateway.

•Administrator username and password—These are used to access the CLI on the server. To simplify management, Cisco recommends that you use the same username and password on all Cisco TelePresence Exchange System servers.

•Security password—You must use the same security password that is defined on all of the other Cisco TelePresence Exchange System servers. The database server uses this password to authenticate data requests from the administration and call engine servers.

•Network and access information for the integrated management module (IMM) interface, which is required to implement active/standby redundancy for the database servers, and which enables remote control of the individual database server:

–Location—typically the building, floor, and rack in which the server is installed.

–State and Country—where the server is located.

Use the following guidelines to determine each entry for generating LSCs:

–Refer to your company guidelines for format and entry requirements.

–Supported characters include alphanumeric, space, and the following special characters: .,-_:;{}()[]#.

–Each field supports up to 255 characters.

•IP addresses, hostnames, or pool names for external Network Time Protocol (NTP) clocking sources—You must configure the same NTP entries that are defined on all of the other Cisco TelePresence Exchange System servers.

Installing the Software on and Synchronizing the Replacement for the Initial Secondary Database Server

Caution This procedure will temporarily interrupt MySQL service. Cisco recommends that you complete this task during a maintenance window. During the MySQL service interruption, new calls will not be able to connect to meetings, and users will not be able to schedule meetings.

Installing the Software on and Synchronizing the Replacement for the Initial Primary Database Server

Caution This procedure will temporarily interrupt MySQL service. Cisco recommends that you complete this task during a maintenance window. During the MySQL service interruption, new calls will not be able to connect to meetings, and users will not be able to schedule meetings.

Replacing an Administration or Call Engine Server

Step 1 Obtain the Cisco TelePresence Exchange System installation DVD, or download the software from the following URL and burn the disk image onto a DVD: http://www.cisco.com/go/ctx-download.

Note Make sure that the software version on the installation DVD is the same as the version that is currently running on the peer server of the same role. If you want to upgrade the software, you may do so after you successfully replace the failed server.

If you cannot find your completed worksheet, or if the information has become obsolete, gather the following information for the server that you need to replace:

•Hostname

•IP address and subnet mask

•Default gateway

•Administrator username and password—These are used to access the CLI on the server. To simplify management, Cisco recommends that you use the same username and password on all Cisco TelePresence Exchange System servers.

•Security password—You must use the same security password that is defined on all of the other Cisco TelePresence Exchange System servers. The database server uses this password to authenticate data requests from the administration and call engine servers.