5.6.1. Recover a failed slave

A slave that has failed but which has become available again can be
recovered back into slave mode using the
recover command:

[LOGICAL:EXPERT] /alpha > recover
FOUND PHYSICAL DATASOURCE TO RECOVER: 'host2@alpha'
VERIFYING THAT WE CAN CONNECT TO DATA SERVER 'host2'
DATA SERVER 'host2' IS NOW AVAILABLE FOR CONNECTIONS
RECOVERING 'host2@alpha' TO A SLAVE USING 'host1@alpha' AS THE MASTER
DataSource 'host2' is now OFFLINE
RECOVERY OF DATA SERVICE 'alpha' SUCCEEDED
RECOVERED 1 DATA SOURCES IN SERVICE 'alpha'

The recover command will attempt to recover all the slave resources in
the cluster, bringing them all online and back into service. The command
operates on all shunned or failed slaves, and only works if there is an
active master available.

To recover a single datasource back into the dataservice, use the
explicit form:

[LOGICAL:EXPERT] /alpha > datasource host1 recover
VERIFYING THAT WE CAN CONNECT TO DATA SERVER 'host1'
DATA SERVER 'host1' IS NOW AVAILABLE FOR CONNECTIONS
RECOVERING 'host1@alpha' TO A SLAVE USING 'host2@alpha' AS THE MASTER
RECOVERY OF 'host1@alpha' WAS SUCCESSFUL

In some cases, the datasource may show as
ONLINE and the
recover command does not bring the
datasource online, particularly with the following error:

The datasource 'host1' is not FAILED or SHUNNED and cannot be recovered.

Checking the datasource status in cctrl the
replicator service has failed, but the datasource shows as online:

Select the active slave within the dataservice that you want to use to
reprovision the failed slave. You may use the master but this will
impact performance on that host. If you use MyISAM tables the operation
will create some locking in order to get a consistent snapshot.

5.6.1.3. Slave Datasource Extended Recovery

If the current slave will not recover, but the replicator state and
sequence number are valid, the slave is pointing to the wrong master,
or still mistakenly has the master role when it should be a slave,
then the slave can be forced back into the slave state.

For example, in the output from
ls in cctrl
below, host2 is mistakenly
identified as the master, even though
host1 is correctly operating as
the master.

Datasource host2 should now be
back in the dataservice as a working datasource.

Similar processes can be used to force a datasource back into the
master role if a switch or
recover operation failed to set the role properly.

If the recover command fails,
there are a number of solutions that may bring the dataservice back to
the normal operational state. The exact method will depend on whether
there are other active slaves (from which a backup can be taken) or
recent backups of the slave are available, and the reasons for the
original failure. Some potential solutions include