Navigation

RBD images can be asynchronously mirrored between two Ceph clusters. This
capability uses the RBD journaling image feature to ensure crash-consistent
replication between clusters. Mirroring is configured on a per-pool basis
within peer clusters and can be configured to automatically mirror all
images within a pool or only a specific subset of images. Mirroring is
configured using the rbd command. The rbd-mirror daemon is responsible
for pulling image updates from the remote, peer cluster and applying them to
the image within the local cluster.

Note

RBD mirroring requires the Ceph Jewel release or later.

Important

To use RBD mirroring, you must have two Ceph clusters, each
running the rbd-mirror daemon.

The following procedures demonstrate how to perform the basic administrative
tasks to configure mirroring using the rbd command. Mirroring is
configured on a per-pool basis within the Ceph clusters.

The pool configuration steps should be performed on both peer clusters. These
procedures assume two clusters, named “local” and “remote”, are accessible from
a single host for clarity.

See the rbd manpage for additional details of how to connect to different
Ceph clusters.

Note

The cluster name in the following examples corresponds to a Ceph
configuration file of the same name (e.g. /etc/ceph/remote.conf). See the
ceph-conf documentation for how to configure multiple clusters.

In order for the rbd-mirror daemon to discover its peer cluster, the peer
needs to be registered to the pool. To add a mirroring peer Ceph cluster with
rbd, specify the mirrorpoolpeeradd command, the pool name, and a
cluster specification:

Unlike pool configuration, image configuration only needs to be performed against
a single mirroring peer Ceph cluster.

Mirrored RBD images are designated as either primary or non-primary. This is a
property of the image and not the pool. Images that are designated as
non-primary cannot be modified.

Images are automatically promoted to primary when mirroring is first enabled on
an image (either implicitly if the pool mirror mode was pool and the image
has the journaling image feature enabled, or explicitly enabled by the
rbd command).

RBD mirroring uses the RBD journaling feature to ensure that the replicated
image always remains crash-consistent. Before an image can be mirrored to
a peer cluster, the journaling feature must be enabled. The feature can be
enabled at image creation time by providing the
--image-featureexclusive-lock,journaling option to the rbd command.

Alternatively, the journaling feature can be dynamically enabled on
pre-existing RBD images. To enable journaling with rbd, specify
the featureenable command, the pool and image name, and the feature name:

rbdfeatureenable{pool-name}/{image-name}{feature-name}

For example:

rbd--clusterlocalfeatureenableimage-pool/image-1journaling

Note

The journaling feature is dependent on the exclusive-lock feature. If
the exclusive-lock feature is not already enabled, it should be enabled prior
to enabling the journaling feature.

Tip

You can enable journaling on all new images by default by adding
rbddefaultfeatures=125 to your Ceph configuration file.

If the mirroring is configured in image mode for the image’s pool, then it
is necessary to explicitly enable mirroring for each image within the pool.
To enable mirroring for a specific image with rbd, specify the
mirrorimageenable command along with the pool and image name:

In a failover scenario where the primary designation needs to be moved to the
image in the peer Ceph cluster, access to the primary image should be stopped
(e.g. power down the VM or remove the associated drive from a VM), demote the
current primary image, promote the new primary image, and resume access to the
image on the alternate cluster.

Note

RBD only provides the necessary tools to facilitate an orderly
failover of an image. An external mechanism is required to coordinate the
full failover process (e.g. closing the image before demotion).

To demote an image to non-primary with rbd, specify the
mirrorimagedemote command along with the pool and image name:

rbdmirrorimagedemote{pool-name}/{image-name}

For example:

rbd--clusterlocalmirrorimagedemoteimage-pool/image-1

To promote an image to primary with rbd, specify the mirrorimagepromote
command along with the pool and image name:

rbdmirrorimagepromote{pool-name}/{image-name}

For example:

rbd--clusterremotemirrorimagepromoteimage-pool/image-1

Tip

Since the primary / non-primary status is per-image, it is possible to
have two clusters split the IO load and stage failover / failback.

Note

Promotion can be forced using the --force option. Forced
promotion is needed when the demotion cannot be propagated to the peer
Ceph cluster (e.g. Ceph cluster failure, communication outage). This will
result in a split-brain scenario between the two peers and the image will no
longer be in-sync until a force resync command is issued.

If a split-brain event is detected by the rbd-mirror daemon, it will not
attempt to mirror the affected image until corrected. To resume mirroring for an
image, first demote the image determined to be out-of-date and then request a
resync to the primary image. To request an image resync with rbd, specify the
mirrorimageresync command along with the pool and image name:

rbdmirrorimageresync{pool-name}/{image-name}

For example:

rbdmirrorimageresyncimage-pool/image-1

Note

The rbd command only flags the image as requiring a resync. The
local cluster’s rbd-mirror daemon process is responsible for performing
the resync asynchronously.

The two rbd-mirror daemons are responsible for watching image journals on the
remote, peer cluster and replaying the journal events against the local
cluster. The RBD image journaling feature records all modifications to the
image in the order they occur. This ensures that a crash-consistent mirror of
the remote image is available locally.

The rbd-mirror daemon is available within the optional rbd-mirror
distribution package.

Important

Each rbd-mirror daemon requires the ability to connect
to both clusters simultaneously.

Warning

Only run a single rbd-mirror daemon per Ceph cluster. A
future Ceph release will add support for horizontal scale-out of the
rbd-mirror daemon.