Munin performance boost

Munin is a great resource monitoring tool. However, it can easily max out the server resource depending on number of monitored devices.

I’ve ran into some disk I/O contention issue on my Munin server. I don’t even have that many monitored devices… only eighteen.

There are couple ways to speed up Munin. With the combination of both greatly reduces server load… especially disk I/O.

rrdcached

munin-async

rrdcached will queue RRD changes and flush changes periodically controlled by cron job. This prevents lots of small random writes to disks.

munin-async basically process munin-node data on client itself instead of on server. Server connects to the client and fetch data.

rrdcached:

Install rrdcached on Debian 9.

apt-get install rrdcached

The easiest way is run rrdcached is as the munin user. This requires no files/folders permission modification for existing Munin data. For my use I only utilize rrdcached for Munin use so this is how I set it up.

Restart rrdcached and check /var/run/rrdcached.sock file. This socket file should have group munin and not root.

Ensure rrdcached_socket in /etc/munin/munin.conf point to correct rrdcached sock file.

rrdcached_socket /var/run/rrdcached.sock

munin-async:

Install munin-async on Munin client (Debian 9). Once installed data in /var/lib/munin-async/ will start to populate.

apt-get install munin-async

munin-async communicates over SSH so we’ll need to generate SSH keys for user “munin” on the server and paste into ~munin-async/.ssh/authorized_keys on the client.

Change the default shell /bin/false for munin user on server to /bin/bash.

chsh -s /bin/bash munin

su into munin user and generate SSH keys.
(When generating the SSH keys just press ENTER bunch times to accept the default.)

su - munin
ssh-keygen -t rsa

Now copy content of server’s ~munin/.ssh/id_rsa.pub file and paste into ~munin-async/.ssh/authorized_keys on the client. Prefix the line before “ssh-rsa” so looks something like following. The prefix sets security on SSH logins that only allows running that munin-async command.

While you are still su-ed as the munin user on server you want to SSH to the client at least once, accept the initial connection handshake, and make sure you can get in password-less. You should see some output like following when you do this.