netdata is a system for distributed real-time performance and health monitoring. You can use syslog-ng to collect and filter data provided by netdata and then send it to Elasticsearch for long-term storage and analysis. The aim is to send both metrics and logs to an Elasticsearch instance, and then access it via Kibana. You could also use Grafana for visualization, but that is not covered in this blog post.

I would like to thank here Fabien Wernli for his help in writing this HowTo.

Before you begin

This workflow uses two servers. Server A is the application server where metrics and logs are collected and sent to Server B, which hosts the Elasticsearch and Kibana instances. I use CentOS 7 in my examples but steps should be fairly similar on other platforms, even if package management commands and repository names are different.

In the example command lines and configurations, servers will be referred to by the names “servera” and “serverb”. Replace them with their IP addresses or real host names to reflect your environment.

Installation of applications

First we install all necessary applications. Once all components are up and running, we will configure them to work nicely together.

Installation of Server A

Server A runs netdata and syslog-ng. As netdata is quite a new product and develops quickly, it is not yet available in official distribution package repositories. There is a pre-built generic binary available, but installing from source is easy.

Change to the netdata directory, and start the installation script as root:

cd netdata
./netdata-installer.sh

When prompted, hit Enter to continue.

The installer script not only compiles netdata but also starts it and configures systemd to start netdata automatically. When installation completes, the installer script also prints information about how to access, stop, or uninstall the application.

By default, the web server of netdata listens on all interfaces on port 19999. You can test it at http://servera:19999/.

This configuration sends netdata metrics and also all syslog messages to Elasticsearch directly.

If you want to collect some of the logs locally as well, keep some of the original configuration accordingly, or write your own rules.

You need to change the server name from ServerB to something matching your environment. The f_netdata filter shows one possible way of filtering netdata metrics before storing to Elasticsearch. Adopt it to your environment.

Next, configure netdata. Open your configuration (/etc/netdata/netdata.conf), and replace the [backend] statement with the following snippet:

The “send charts matching” setting here serves a similar role as “f_netdata” in the syslog-ng configuration. You can use either of them, but syslog-ng provides more flexibility.

Finally, restart both netdata and syslog-ng so the configurations take effect. Note that if you used the above configuration, you do not see logs arriving in local files anymore. You can check your logs once the Elasticsearch server part is configured.

Configuring Server B

Elasticsearch controls how data is stored and indexed using index templates. The following two templates will ensure netdata and syslog data have the correct settings.

3. You can now edit your Elasticsearch configuration file and enable binding to an external interface so it can receive data from syslog-ng. Open /etc/elasticsearch/elasticsearch.yml and set the network.host parameter:

network.host:
- [serverB_IP]
- 127.0.0.1

Of course, replace [serverB_IP] with the actual IP address.

4. Restart Elasticsearch so the configuration takes effect.

5. Finally, edit your Kibana configuration (/etc/kibana/kibana.yml), and append the following few lines to the file:

Testing

You should now be able to log in to Kibana on port 5601 of Server B. You should set up your indexes on first use, and then you are ready to query your logs. If it does not work, here is a list of possible problems:

“serverb” has not been rewritten to the proper IP address in configurations.

SELinux is running (for testing, “setenforce 0” is enough, but for production, make sure that SELinux is properly configured).

The firewall is blocking network traffic.

Further reading

I gave here only minimal instructions to get started with netdata, syslog-ng, and Elasticsearch. You can learn more on their respective documentation pages:

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or you can even chat with us. For a long list of possibilities, check our contact page at https://syslog-ng.org/contact-us/. On Twitter, I am available as @PCzanik.