Percona XtraDB Cluster: SElinux is not always the culprit !

If you are using SElinux, you should know that it’s advised to disable it to avoid issue with PXC. Generally the communication between your nodes doesn’t work properly and a node having SElinux enabled won’t be able to join the cluster.

So when a node doesn’t join the cluster where it should, my first reflex is to have a look at audit.log. But recently I faced another problem: the node joined the cluster but SST failed (whatever which method was used, discarding skip).

I checked SElinux and it was of course disabled, then I add some debug information in the SST script but it seemed that the script was never launched. And this time the culprit is called : AppArmor !

Percona doesn’t provide any AppArmor profile for PXC, but it seems that on this server (Ubuntu TLS), a previous version of MySQL was installed and then removed but the AppArmor profile was still present.

So if you use apparmor (or if you don’t know) and you want to check is there is a profile for mysql, you can run the following command :

Shell

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

root@testmachine:~# apparmor_status

apparmor module isloaded.

7profiles are loaded.

7profiles are inenforce mode.

/sbin/dhclient

/usr/lib/NetworkManager/nm-dhcp-client.action

/usr/lib/connman/scripts/dhclient-script

<strong>/usr/sbin/mysqld</strong>

/usr/sbin/named

/usr/sbin/ntpd

/usr/sbin/tcpdump

0profiles are incomplain mode.

2processes have profiles defined.

2processes are inenforce mode.

/usr/sbin/named(1205)

/usr/sbin/ntpd(1347)

0processes are incomplain mode.

You can disable a profile easily by running

Shell

1

2

sudo ln-s/etc/apparmor.d/usr.sbin.mysqld/etc/apparmor.d/disable/

sudo apparmor_parser-R/etc/apparmor.d/usr.sbin.mysqld

For more information related to AppArmor, you can refer to Ubuntu’s wiki

So now if you run ubuntu, you have two things to check first : SElinux and AppArmor !

Note: We often advise to disable SElinux and AppArmor on dedicated MySQL servers to avoid the performance overhead

I disabled SELinux everywhere, always have. There are much better, less frustrating means to secure a system against most threats. I really rarely come across a senior engineer/admin who does not turn off SElinux entirely. I suppose if you operate a public SSH server, it’d be useful. My systems operate in much more controlled environments.

SELinux is used more widely than you might think, particularly in industries that are subject to significant governmental regulation, such as healthcare and banking/finance, and even on servers that have no direct connection to the public internet.

I agree that SELinux configuration can be quite frustrating, but I think the learning curve is worthwhile. Even if you don’t make any tweaks besides adjusting the security context to keep your MySQL data files somewhere other than /var/lib/mysql, understanding SELinux gives you an extremely fine-grained level of control over your system, and IMHO, that’s really cool.