Scheduled ElasticSearch index cleanup

9 December 2018

It’s been over a year since I’ve started using ELK stack for logging purpose. In the meantime, I was able to successfully introduce it in a few development teams, totally changing the way we are working with application logs. Everything is working fine and the only downside so far was the need for periodical maintenance work. By maintenance, I mean removing old indices. If the disk free space drops below certain level the ElasticSearch stops working correctly. This behavior is controlled by a number of ElasticSearch parameters described in Disk-based Shard Allocation section. Together with ElasticSearch, Kibana also goes down and when we try to use it we get the error screen with Elasticsearch plugin is read message as below:

In this situation, all Kibana functions are disabled, even the DevTool page. In order to bring it back to life, we have to restart ElasticSearch service and free some disk space by deleting old indices. Because DevTool is not available we have to do this using anything that can issue HTTP POST request (for example curl or Postman). According to Murphy’s law, this always happens when you badly need to check something in the logs, so it would be great to have a mechanism that prevents this kind of ELK downtime.

There is a lesser known tool from Elastic stable called curator which can help with managing ElasticSearch indices. In this post, I will show you how to build a self-maintenance mechanism for ELK stack using curator together with cron scheduler.

Before we install curator we have to update package repository path. Without it apt-get will install curator’s old version. In order to do that, please add the following line to /etc/apt/sources.list file:

After adding this new repository path we have to download the package lists and update them to get information on the newest versions of packages and their dependencies. This can be simply achieved with the command:

Save this content to config.yml. For correct logging, you need to create /var/log/curator directory.
Sometimes using 127.0.0.1 for client.hosts is not working and you need to provide the real IP address (I think it depends on server configuration). For more info about curator configuration please check the official reference.

In my example, I used 0 0 * * * as a time-expression which means that my command will run every day at midnight. If you don’t know crontab time syntax you can easily generate the desired expression using https://crontab-generator.org/. After saving crontab config file, if everything was configured correctly, you should get crontab: installing new crontab message. That’s all. You can verify if everything is working correctly by examining /var/log/curator/curator.log log file.

As you saw, thanks to curator and cron scheduler I was able to create the mechanism for automated ELK maintenance. Now I don’t need to worry anymore that ELK server will run out of free disk space and bring all logging stack down. I would like to thank Robert Łysoń (@RobertLyson) who helped me with preparing curator configuration. Next time the beer is on me ;)

If you find this blog post useful and you think it's worth to share this idea with others, please don't hesitate to use these buttons below: