What this is about

So I have couple of MariaDB servers and wanted to take backups of some of
databases hosted on those servers. I could simply schedule an systemd-timer
job on every of those machine that would simply mysqldump
required databases on regular basis - let's say every our. And then send it
somewhere. Or make external service fetch those backups regularly.

But I don't like this idea. First of all because it increases my RPO,
and secondly - in case one of those DB servers is down I'd have to redeploy it,
load backups etc. It takes time.

So instead of this idea (an old one, mentioned years ago in 2010 in Mysql High Availability)
is to replicate all databases to a slave servers and take backups of all slave
DBs.

However this wasn't easy to perform back then in MySQL, because there was no
such thing as multi - master replication. Basically you can't start replication
from more than one master. In MySQL you had to run separate slave server for
each master you replicated.

And in MariaDB 10 You can finally do this. This image show the idea (I took it
from mariadb.com page):

Ok, how to do this?

In my scenario I had 2 already running MariaDB 10.x servers (from Fedora
upstream) and a fresh one, that I pointed to be the slave for above masters.

Also a downtime for each master is needed for a time of taking full backup. I
didn't find an easy way to get around this. If you do - please leave a
comment - thx!

Last, but not least - I used mariadb-server version 10.1.26.

1. Take backups of both masters.

Should I really write this point? ;)

2. Prepare your masters' configurations

Now we need to make both master servers actual masters :) In order to do that
I edited /etc/my.cnf.d/mariadb-server.cnf (find your own my.cnf file -
probably in /etc/my.cnf) and put there something like this (of course leave
other configuration options that lay there):

log-bin - thanks to this server will become master and start logging
binary log. This means, that a log of changes will be saved to files and later
sent to slaves and replayed there. See this article
for more information.

server_id - basically keep this value unique for every server in your
replication topology. This must be a number.

log-basename - will be used as a part of name for log-bin files

binlog-ignore-db - this tells which databases must not be replicated.
Very useful when you want to replicate all DBs but for some set of system DBs.

gtid-domain-id - basically keep this unique for every master. Here you
can read
details about GTIDs in multi - source replication topologies.

gtid-ignore-duplicates - this is actually not very needed in my
replication topology. Thanks to this setting when slave receives event with
GTID that was already processed it will ignore it. It is possible in situation
when you have a chain of master-slave/master-slave replications.

3. Prepare masters data for sending to slave

Execute this for every master. I did it on both of my servers:

This procedure execution time mostly depends on how long it takes to take
full backup of required databases. You can test it by running time mysqldump --databases db1,db2,db3 -u some_user -p > dbs.sql
beforehand.

Restart Mariadb service in order to apply above configurations (this might
be not needed as probably you can set above setting from mariadb SQL console
during runtime - you can easily find how to do that in the internets; just a
note for people who wanna minimize downtime).

Log into Mariadb SQL console and execute: FLUSH TABLES WITH READ LOCK; -
this will lock and disable all writes to this server. Prior to doing that I
always shutdown HTTP services or put them in maintenance mode. Keep in mind
that also some cron/timers services might try to connect. Thus locking on DB
layer is most save.

Execute select @@gtid_binlog_pos; and save this position for later use.