The structure of automation of Rolling Upgrade process is quite straightforward.
* We need to generate config files(.cnf) for all nodes with respective path and version for MariaDB.
Say we have 5 nodes, then there will be 5 config files – like mynode1_10.3.cnf, mynode2_10.3.cnf etc.
But we need also 10.4 version config files for Upgrade – like mynode1_10.4.cnf, mynode2_10.4.cnf etc.
Main difference between those config files is – path of Galera – it should be 3 for 10.3 and 4 for 10.4
* Nodes need to be started with old 10.3 version of MariaDB and Galera 3, with respective config files mentioned above.
* After successfully cluster start, we need to start shut down and upgrade from last node – for us it is node5
So, shut down node5, start it 10.4 binary append new config(mynode5_10.4.cnf not mynode5_10.3.cnf).
Run mysql_upgrade after successful start with new binary.
Repeat this process in a loop until whole cluster members upgraded as described. So the next node should be node4, node3, node2 and node1 at last.

But there is another question – should be there any load for nodes during upgrade? In other words, should we run DDL/DML after upgrading each node? For eg,
* Upgrade node5, run some load on node1(which is still old version) – see if data replicated from old to new.
* Upgrade node5, run some load on it – see if data replicated from new to old.

Based on these entries, here is the easy way to lose node5: MDEV-18422
Simply, upgrade node5 and then run DDL on node1 get the crash, then be happy – because it is already fixed.

Another quite critical issue was Data Inconsistency.
All nodes already upgraded and Streaming Replication was enabled on each node.
Now it was turn to Create database and then Drop it on node5. Unfortunately it was dropped from all other nodes but not from node5 itself – MDEV-18587
Fixed.

In general we love Segfaults but not in cluster 🙂 Interestingly if you enable log-bin and log-slave-updates on all nodes of cluster and try to do Rolling Upgrade the last upgraded node – node1 will crash – MDEV-18588
Fixed.

Hall of the fame has permanent member called Sysbench. Here is the crash while doing sysbench prepare – MDEV-18631

There are several issues found on the way but are not related directly to Upgrade Tests. But in general Upgrade tests are great playground to get dirty with all sort of issues.
This post likely will be updated as new issues or fixes will come.

Great, now it is much more clear 🙂 Everything should exist in this MySQL version because it is the latest.
So after testing bunch of combinations it turned out it was due to COMPRESSION=’none’ statement.
If you exclude it from create statement:

You can puzzle it + can make a list of combinations to find the exact problem as me.
But it seems to be a bug – I can use encryption=’N’ but can not use compression=’none’?
Reported as -> https://bugs.mysql.com/bug.php?id=90040

In this topic I would like to make some notes about questions, how we can contribute or help on Open Source project.
The first thing is, to learn the product itself, how to install, how to remove, how to use etc.
Of course, the first place for this information is an official documentation.
How you can use the DOC? I am taking following approach:
* Read the doc to learn the needed thing.
* Test the info provided in the DOC –
if the info is valid(i.e you have different results)
continue
if it is not valid:
Report a bug – then continue reading.

The issue can be in DOC itself or in the product.
Did you find small typo in DOC? Did you think some info about topic is wrong? Or maybe you found wrong code samples?
Report it immediately and you will be the part of community.

Let’s take some DOC bug reports and examine them to get the idea.
The first is -> #87545:
While reading Connector/Python X Plugin reference did not see any info about Error module(mysqlx.errors).
So this missing info must be there, that is why reported as described. Simple? But it is crucial.

Another one is -> #81036
There is an entry about:
“Installing MySQL Shell on Yum-based Systems”:

> Install MySQL Shell with this command:

sudo apt-get install mysql-shell

Well you have already noticed that, it must be yum command not apt. Funny? But it is needed.

Here, I saw the error message in 5.7 but read about this only in 8.0 versions DOC. So the same info should be in 5.7s DOC as well, because I see error on 5.7 too.

In conclusion, we should understand the importance of documentation as well as, the importance to report DOC issues.
Many things are happening because of misunderstanding DOC or because of wrong info inside the DOCs.

Think about this twice in the next time, when you find some weird thing in DOC and just ignore it.
It is a real life, the thing which is not crucial for us can be important for others.

Percona Server has implemented this feature to be a lightweight alternative to FLUSH TABLES WITH READ LOCK for both physical and logical backups. Three new statements are now available: LOCK TABLES FOR BACKUP, LOCK BINLOG FOR BACKUP and UNLOCK BINLOG.

So by default, if you are using PS it is going to use LOCK TABLES FOR BACKUP prior copying non-InnoDB tables.

Well, how to force to use FTWRL instead of Backup Locks?
Congratulations – we have --no-backup-locks option, if you specify it, the FTWRL will be used.

* First we need some MyISAM tables. I used sysbench to create some tables and then altered them:

# Altering some of the table engines from innodb to myisam
for i in range(20, 25):
sql_alter_engine = "alter table sysbench_test_db.sbtest{} engine=myisam".format(i)
RunBenchmark.run_sql_statement(basedir=self.basedir, sql_statement=sql_alter_engine)

* As I am using the MySQL client directly, the best is to have some bash script for running SQL redirected to /dev/null(because I don’t need any result from SQL result, I will run benchmark query):

with concurrent.futures.ProcessPoolExecutor(max_workers=50) as pool:
for _ in range(10):
for i in range(20, 25):
pool.submit(
self.parallel_sleep_queries(basedir=self.basedir,
sock="{}/socket.sock".format(self.basedir),
sql="select benchmark(9999999, md5(c)) from sysbench_test_db.sbtest{}".format(
i)))
self.all_backup() # Taking backup with options provided above

So the thing is quite simple here, I have created a bash script to call mysql cli, then called it from Python method with Popen and the called this method concurrently.
The result is quite handy. Actual backup command(it is generated from config file):

$ /home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/bin/mysql -urepl --password='Baku12345' --port=10023 --socket=/home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/socket.sock
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 623
Server version: 5.6.37-82.2-debug-log MySQL Community Server (GPL)

So it turned out, the blank users were preventing slave connection to master server 🙂