Infrastructure, Linux, Monitoring, Automation and ITSM

Sunday, June 1, 2014

Graphite it an opensource application used to track metrics on a time series database. You can send metrics using almost any programming language and its way easier than other good options that may result a little more difficult to understand tools like OpenTSDB and RRDtool.

There are lots of tutorials and scripts that show how to install it, most of them use apache to deploy the web interface. The graphite web app is written in Django and as such it's really easy to deploy using nginx.

Check http://yourIPAddress:8080 and you should see the graphite web interface.

If you have any issues, remember to check the following log files for possible errors:

/var/log/nginx/graphite.error.log
/var/log/uwsgi/app/graphite.log

As a last resource you can set the Django app into debug mode by setting "Debug = True" on "/opt/graphite/webapp/graphite/local_settings.py".It should provide enough information to give you a hint on what may be causing the problem.

Installing runit

After installing, you'll have to options to start using runit: reboot o reread your /etc/inittab file by running:

init q

From now on, every time we start your OS it will run the runsvdir daemon. This process will be responsible for monitoring the /etc/service/ directory where we will configure our services. By each service running on this directory, the runsvdir daemon will spawn a new runsv process.

Configuring our first service.

Create the service directory

mkdir -p /etc/sv/test

Create the service script and the script to log everything that happens with the service:

touch /etc/sv/test/run /etc/sv/test/log/run

Give running persmission. This is very important. Otherwise the services won't run:

chmod u+x /etc/sv/test/run

Our really simple 'service' that writes to stdout and stderr every and die after 5 seconds:

#!/bin/sh
exec 2>&1
exec bash - <

Note: We need to use exec in order to make our command replace the current shell without creating a new process.
Content of /etc/sv/test/log/run:

#!/bin/sh
exec chpst -u nobody svlogd -ttt /var/log/test/

We have our services configured but not under supervision. We need to create a symbolic link to the /etc/service/ directory:

Tuesday, September 3, 2013

The situation:

This may sound curious, but yes I had to simulate a filesystem with no space left in order to reproduce a failure that caused problems with the uploads on an PHP application. The upload tmp dir got full and the application stopped uploading files.

The application was running on a VM and there was no LVM. This meant that I would had to add a disk to the VM, restart de VM, create a new partition... It was to many steps for a simple test. Besides that, I wouldn't use the partition again after the test.

The Solution:

Create a filesystem on a file, use it as a loopback device and fill the filesystem. I thought loopback devices were useful just for mounting ISOs and disk images, but no, they became really useful on this situation:
That's it. I updated the upload_tmp_dir PHP parameter to:
upload_tmp_dir = /mnt/php-full-device

The developer corrected the bug and I didn't have add another disk to the VM (I know, I'm lazy).

Bonus Track:

Here I leave some additional commands that might be useful:
As you may appreciate this is easy to implement and doesn't require any server restarts.