Developer Community

Developer Blog

Using logstash to Push Metrics to Graphite

One of the cool things we do on the Cloud Databases operations side of the house is come up with statistics that can help us gain insight to hardware performance to identify issues with systems. We use some really cool tools, but one of the most versatile tools we work with is logstash. The goal of this article is to get you started pushing metrics with logstash that you may already collect to Graphite. Along the way, I'll be showing you how to get started with logstash, test your configuration locally and then start pushing your first metrics to Graphite with some different examples along the way.<!-- more -->

Openjdk or a form of modern java installed

Sysstat package installed

A functional Graphite installation

In this tutorial, our carbon cache listening endpoint for Graphite is 10.10.10.10:2003. For help installing Graphite, see the links section below. Our host running logstash is going to be named "MYHOST." We were collecting system statistics with Sysstat and we decided to leverage the fact that we already have metrics being collected.

To get started using logstash, you'll want to start by grabbing the monolithic jar file:

This contains the logstash code and all the plugins provided by the base logstash configuration. You can extend logstash with Ruby plugins too, but we'll cover that some other time. The logstash configuration has three possible sections: input, filter and output. Here, we'll explore input types exec; filter types split and grok; and the Graphite output filter.

Back in the early logstash days, I got started testing my configuration by making a wrapper script. As it turns out, this was very similar to the guide to getting started with logstash that you'll find in the links section at the bottom of this post, but they cover far more functionality that we're not concerned with. So with that said, this is the abbreviated version of that getting started guide, but with a wrapper script for testing; it reduces typing and ensures repeatability.

First, let’s set up our logstash environment. Let’s create a directory called "logstash" in /opt, download the monolithic jar, and create a couple of files that help with the testing. To the code!

Since extra GNU utility calls are cheap and executed in the same subshell, we can manipulate any output into something we can easily push through logstash. Here, we only output the load average stats found in /proc/loadavg with the awk statement. Note: in our input section we set a logstash "type" which sets the @type variable. The logstash type and tag fields let you target specific sets of data with your filter and output sections. Next, the grok filter plugin is used to describe common patterns in unstructured data using the "%{SYNTAX:SEMANTIC}" form. The “syntax” is the type of data you're grabbing (ex: NUMBER, INT, WORD) and “semantic” is the name of the field. The namedcapturesonly parameter makes the grok plugin only store the items we specified to capture instead of any extra items that grok may naturally collect. Grok is ultra powerful and shines when you're pulling data from logs, however, in this case we're using it to grab our one-minute, five-minute and 15-minute load averages.

Let’s run it with the new config; hit CTRL+C to break out of the shell script and stop running logstash:

The example config file is exactly the same as before with the exception of the output filter; now it's pushing data to our Graphite endpoint. We're grabbing the "system-loadavg" type, and we are creating an array of metrics. We use a built-in variable @sourcehost set by logstash to set up the Graphite metric scope for this particular host - notice that we're only setting the metric scope and value. Then logstash will use the @timestamp when it pushes out each metric pair to Graphite. Assuming your Graphite install is functional, we could then pull data from our URL: http://10.10.10.10:9000/render/?target=hosts.MYHOST.loadavg.1m

Some of the tools we use for system metrics are sar and sysstat. We track things like disk performance and network performance. Here's and example configuration file that logs network statistics from sar:

The input section is the same concept as before but we're running a sar report and pruning the extra stuff like column headers. The filter section first passes our system-netstats through the split filter – this splits common multiline data and hands each line through the logstash data pipeline individually. The grok filter is then used to name each field in the input, and the output section creates an array of scope and value data pairs for our graphite endpoint. We can pull one of the stats collected for device eth0 with the URL: http://10.10.10.10:9000/render/?target=hosts.MYHOST.netstats.interfaces.eth0.rxpck_s

As you can see, logstash has some great functionality that allows us to grab data and pass it on to Graphite. Using GNU utilities, you can leverage existing data collection through daemons such as sysstat and start collecting some great statistics and generate graphs with Graphite.