In the Graphite Series blog posts, I'll provide a guide to help through all of the steps involved in setting up a monitoring and alerting system using a Graphite stack. Disclaimer: I am no expert, I am just trying to help the Graphite community by providing more detailed documentation. If there's something wrong, please comment below or drop me an email at feangulo@yaipan.com.

Graphite Back-end

Graphite is composed of multiple back-end and front-end components. The back-end components are used to store numeric time-series data. The front-end components are used to retrieve the metric data and optionally render graphs. In this blog post, I'll focus on the back-end components: Carbon and Whisper.

Metrics can be published to a load balancer or directly to a Carbon process. The Carbon process interacts with the Whisper database library to store the time-series data to the filesystem.

Install Carbon

Carbon is a daemon that listens for time-series data using an event-driven networking engine called Twisted. The Twisted framework permits Carbon processes to handle a large number of clients and handle a large amount of traffic with a low amount of overhead.

Client applications publish metrics by sending data points to a Carbon process. The application establishes a TCP connection on the Carbon process' port and sends data points in a simple plaintext format. In our example, the port is 2003. The TCP connection may remain open and reused as many times as necessary. The Carbon process listens for incoming data but does not send any response back to the client.

The datapoint format is defined as:

a single line of text per data point

a dotted metric name at position 0

a value at position 1

a Unix Epoch timestamp at position 2

spaces for the position separators

For example, here are some valid datapoints:

carbon.agents.graphite-tutorial.metricsReceived 28198 1400509108

carbon.agents.graphite-tutorial.creates 8 1400509110

PRODUCTION.host.graphite-tutorial.responseTime.p95 0.10 1400509112

Client applications have multiple ways to publish metrics:

using the plaintext protocol with tools such as the netcat (nc) command

using the pickle protocol

using the Advanced Message Queueing Protocol (AMQP)

using libraries such as the Coda Hale metrics library

For simplicity, in this tutorial I'll be using the plaintext protocol through the netcat command. To publish the example datapoints listed above, run the following commands:

Aggregation method, max retention, xFilesFactor, and all of the other attributes of the Whisper file are important to understand. Don't worry if you're lost at this point, I'll be covering these in more detail in the next blog post.