17.4.8 Switching Masters During Failover

You can tell a slave to change to a new master using the
CHANGE MASTER TO statement. The
slave does not check whether the databases on the master are
compatible with those on the slave; it simply begins reading and
executing events from the specified coordinates in the new
master's binary log. In a failover situation, all the servers
in the group are typically executing the same events from the same
binary log file, so changing the source of the events should not
affect the structure or integrity of the database, provided that
you exercise care in making the change.

In this diagram, the MySQL Master holds the
master database, the MySQL Slave hosts are
replication slaves, and the Web Client machines
are issuing database reads and writes. Web clients that issue only
reads (and would normally be connected to the slaves) are not
shown, as they do not need to switch to a new server in the event
of failure. For a more detailed example of a read/write scale-out
replication structure, see
Section 17.4.5, “Using Replication for Scale-Out”.

Each MySQL Slave (Slave 1, Slave
2, and Slave 3) is a slave running
with binary logging enabled, and with
--log-slave-updates=OFF. Because
updates received by a slave from the master are not logged in the
binary log when
--log-slave-updates=OFF is
specified, the binary log on each slave is empty initially. If for
some reason MySQL Master becomes unavailable,
you can pick one of the slaves to become the new master. For
example, if you pick Slave 1, all Web
Clients should be redirected to Slave
1, which writes the updates to its binary log.
Slave 2 and Slave 3 should
then replicate from Slave 1.

The reason for running the slave with
--log-slave-updates=OFF is to
prevent slaves from receiving updates twice in case you cause one
of the slaves to become the new master. If Slave
1 has --log-slave-updates
enabled, which is the default, it writes any updates that it
receives from Master in its own binary log.
This means that, when Slave 2 changes from
Master to Slave 1 as its
master, it may receive updates from Slave 1
that it has already received from Master.

Make sure that all slaves have processed any statements in their
relay log. On each slave, issue STOP SLAVE
IO_THREAD, then check the output of
SHOW PROCESSLIST until you see
Has read all relay log. When this is true for
all slaves, they can be reconfigured to the new setup. On the
slave Slave 1 being promoted to become the
master, issue STOP SLAVE and
RESET MASTER.

On the other slaves Slave 2 and Slave
3, use STOP SLAVE and
CHANGE MASTER TO MASTER_HOST='Slave1' (where
'Slave1' represents the real host name of
Slave 1). To use CHANGE MASTER
TO, add all information about how to connect to
Slave 1 from Slave 2 or
Slave 3 (user,
password,
port). When issuing the CHANGE
MASTER TO statement in this, there is no need to specify
the name of the Slave 1 binary log file or log
position to read from, since the first binary log file and
position 4, are the defaults. Finally, execute
START SLAVE on Slave
2 and Slave 3.

Once the new replication setup is in place, you need to tell each
Web Client to direct its statements to
Slave 1. From that point on, all update
statements sent by Web Client to Slave
1 are written to the binary log of Slave
1, which then contains every update statement sent to
Slave 1 since Master died.

When Master becomes available again, you should
make it a slave of Slave 1. To do this, issue
on Master the same CHANGE
MASTER TO statement as that issued on Slave
2 and Slave 3 previously.
Master then becomes a slave of S1ave
1 and picks up the Web Client writes
that it missed while it was offline.

To make Master a master again, use the
preceding procedure as if Slave 1 was
unavailable and Master was to be the new
master. During this procedure, do not forget to run
RESET MASTER on
Master before making Slave
1, Slave 2, and Slave
3 slaves of Master. If you fail to do
this, the slaves may pick up stale writes from the Web
Client applications dating from before the point at
which Master became unavailable.

You should be aware that there is no synchronization between
slaves, even when they share the same master, and thus some slaves
might be considerably ahead of others. This means that in some
cases the procedure outlined in the previous example might not
work as expected. In practice, however, relay logs on all slaves
should be relatively close together.

One way to keep applications informed about the location of the
master is to have a dynamic DNS entry for the master. With
bind you can use nsupdate
to update the DNS dynamically.