When using and tending an Openshift Origin cluster (for example, Origin version 3.7), it is normal to start the storage allocation in small sizes. However soon we find that storage for registry get filled up quickly with images from each build process. This post will show how to clean them up.

Preparation before pruning

First you need oc (origin client) binary and a user account with cluster administration capability.

If the openshift docker registry is installed inside the cluster without external access, then you also going to need OS access to one of the hosts inside the cluster.

First step is to login to the cluster from your client or inside one of the hosts:

oc login

Prune steps

Reading the documentation (https://docs.openshift.com/enterprise/3.0/admin_guide/pruning_resources.html) we find that the pruning starts at deployment, then builds, and last images.

Pruning Deployment

Run this to preview which deployment are going to be pruned:

oc adm prune deployments

Then execute the pruning :

oc adm prune deployments --confirm

We could use the CLI utility oadm (origin adm) or oc adm command, depending on availability of the executable.

Pruning Builds

Run this to preview the builds :

oc adm prune builds

or

oadm prune builds

Then execute the pruning :

oc adm prune builds --confirm

Pruning Registry Images

And finally, we prune images. Images could not be pruned if there are some deployments are referring to the image, thus the prune deployment and build steps above are need to be done first.

oadm prune images

Confirm the pruning with additional --confirm flag:

oadm prune images --confirm

If we the registry is not accessible, we get this message :error: error communicating with registry: Get http://IPredacted:5000/healthz: dial tcp IPredacted:5000: getsockopt: operation timed out
Such error means we need to ssh into one of the hosts in to be able to prune images.

I just recently upgraded a database server that were previously running MySQL 5.1 (standard, Sun/Oracle version) into Percona Server 5.7. A few quirks notable enough to warrant this blog post.

Planning and Preparation

A Percona blog post (mysql-upgrade-best-practices) stated that the best way to upgrade with such huge difference in major version (5.1 to 5.7) is to do a full logical dump for all database except mysql, dump user and grants, uninstall database and remove datafiles, then install new version and import the logical dump and grants. But alas the database we are going to upgrade is so big and the IO subsystem became some sort of bottleneck when doing logical dump, our colleagues tried to do mysqldump and it tooks more than 2 days to run, prompting us to cancel the backup (otherwise it would interfere with workday application usage of the database). Reading the blog I noted that :

for major version upgrade, using logical dump and restore dump is the safest way to go.

for minor version upgrade, in-place upgrade is possible.

for major version upgrade, do not skip versions. I deduced that the sequence is : 5.1 ⇨ 5.5 ⇨ 5.6 ⇨ 5.7

The main thing is to be careful. In IT world, to be careful is to do backups before you do something big.

Backing up is mandatory for any upgrade task relating to production system. Our database server is running on VMWare platform, so we prepared to do a snapshot before doing the upgrade. But alternative backup is in order, so we installed Percona Xtrabackup to create such backup. Another option is to use mydumper (see https://www.percona.com/blog/2015/11/12/logical-mysql-backup-tool-mydumper-0-9-1-now-available/), but in the past I were having problems compiling it so I avoid it.

Percona Xtrabackup

Being an open-source software, it is strange that the Percona Xtrabackup PDF manual is not easily found. Need to register your name, address, company, et cetera, just to get a hold of the PDF version of the manual. Anyway, the quirk is, sometimes the manual said 'innobackupex' and sometimes said 'xtrabackup'. Checking the installed file reveals that innobackupex is symbolic-link to xtrabackup.. seems that the two are now interchangeable, but no such statement found in the PDF manual.

Quoting the Percona Xtrabackup 2.4.8 PDF manual in chapter Ten :

innobackupex is the tool which provides functionality to backup a whole MySQL database instance using the
xtrabackup in combination with tools like xbstream and xbcrypt.

The paragraph above is confusing, because seems that the correct way to do backup is by using innobackupex. But in the web version of the manual we get :

innobackupexinnobackupex is the symlink for xtrabackup. innobackupex still supports all features and syntax as 2.2 version did, but is now deprecated and will be removed in next major release.

Seems that the web version is the best one to follow, unfortunately I noticed this after completing the backup task.

One quirk is xtrabackup doesn't like the internal innodb engine (error : Built-In InnoDB 5.1 is not supported in this release), so I need to add these in /etc/my.cnf and restart the db before doing the xtrabackup command :

ignore-builtin-innodb
plugin-load=innodb=ha_innodb_plugin.so

Replacing MySQL 5.1 with Percona Server 5.1

To replace mysql 5.1, we uninstall them and then install percona server. Both (mysql and percona server) are using my.cnf. The steps are :

stop mysqld service (note: mysqld is the Oracle-based service, percona-based service in RHEL is mysql)

These steps quite straightforward, but Percona Server 5.1 don't like the innodb engine plugin that were installed before (Error: The option ignore-builtin-innodb is incompatible with Percona Server with XtraDB), forcing me to remove/comment these two lines :

This error pops up when I tried to run mysql_upgrade. Seems this is a known bug (https://bugs.mysql.com/bug.php?id=72896) that didn't get fixed, in which mysql_upgrade calls flush privileges which rereads grant tables . Our solution is to use alternate syntax to execute mysql_upgrade :

mysql_upgrade -u root -p

After doing that, all is well.

Upgrading Percona Server 5.6 to 5.7

The reference for the 5.6 ⇨ 5.7 upgrade is https://www.percona.com/doc/percona-server/5.7/upgrading_guide_56_57.html. Which essentially the same with 5.5 ⇨ 5.6 upgrade, so I would not duplicate here. But there is a few major difference :

By default, mysql_upgrade will convert tables that are using date,time, and timestamp to the MySQL 5.6.4 format (note that the 5.6 upgrade process does not issue a warning about these tables, and doesn't suggest conversion process either). The new binary date/time format are more space-efficient and allows extension types (such as TIMESTAMP(4)) with fractional seconds. Refer to https://www.percona.com/blog/2016/04/27/upgrading-to-mysql-5-7-focusing-on-temporal-types/, the majority running time of mysql_upgrade are now spent converting these tables (shown as ALTER TABLE ... FORCE). A workaround to prevent this is to run mysql_upgrade with -s flag / --upgrade-system-tables flag, which does not upgrade data tables.

Default sql_mode is now includes ONLY_FULL_GROUP_BY, STRICT_TRANS_TABLES. The only_full_group_by now fixes enforces group by to be in proper form, but unfortunately this breaks many sloppy SQLs that previously allowed to run. The strict_trans_tables changes the mysql behavior on INSERT and UPDATE queries regarding invalid field contents.

Conclusion

In-place upgrades from 5.1 to 5.7 are indeed feasible, especially when the data size is quite large and mysqldump takes too much time to run. For backups, Percona Xtrabackup will perform the task with better speed than mysqldump (but with binary backup as a result).