because every new challenge hides an opportunity

…and promise_summary.log met Solarized

Inspired by the blog post at nanard.org, I’ve spent a few days hacking on promise_summary.log and rrdtool, and I have finally something to show.

Let’s recap quickly: a file promise_summary.log in cfengine’s workdir (usually /var/cfengine) contains a summary of every agent run: when the run started and finished, which version of the policy has run, and how many promises (in percentage) were kept, repaired, and not repaired. The first thing I wanted were graphs for these metrics; a medium-term goal is to bring these metrics into some well known tool like Munin or Cacti — that will come later.

all aspects of a graph are customizable, and different type of graphs can be embedded in the same picture

I had previous experience with RRDtool, and I knew the downsides of course, mainly: the odd, cryptic syntax. What I had forgotten since such a long time was that it’s actually easier than it looks 🙂 …

Create the round-robin database
Since we are going to use RRDtool, the first step is to create a round-robin database (an RRD file). When one runs the following shell script, create_rrd.sh, providing the path for promise_summary.log as an argument, it will create the appropriate RRD file:

The first line reads the first timestamp in the first line of the log, subtracts 10 seconds from it, and assigns the result to the variable START. That value is then used as a starting timestamp for the RRD. The step in which we collect the samples is 300 seconds (the normal interval between two runs of an agent).

We create nine round-robin archives (RRA) in the RRD:

the first three are created so that it can save 12 points per hour for 7 days, so: 12*24*7=2016 points;

the second group will save 2 points per hour for 40 days, so: 2*24*40=1920 points;

the third group will save 1 point every 4 hours for 400 days, so: 0.25*24*400=2400 points;

we save the average, the minimum, and the maximum of the values collected in an interval

Load data into the RRD
The second step is to parse the log, and load the data into the RRD. This is Perl’s job, and that’s parse_summary.pl. This script takes the path for parse_summary.log as an argument, and prints the data in a format suitable to rrdtool update:

Graphing the data
Now that the data is into the RRD, it’s time for graphs. Many geeks show a rather bad taste when choosing graph colors, and I’m no exception, so rather than trying to choose them myself, I referred to the wonderful Solarized work.

The percentage of kept, repaired, and not repaired promises should always sum up to 100 (except when rrdtool rounds off the numbers a bit perhaps), so they are good candidates for a stacked area graph. On top of them, I’ll put a line graph of the duration. For the canvas, I chose the base light colour (base3), and the promise percentage got the three darker base colours following base3 for themselves (base2, base1, and base0 for kept, repaired, and not repaired respectively). The major grid lines, usually red, got the red colour from the Solarized palette, and the minor grid lines got base00. Finally, the duration line gets the blue colour.

Before getting to the meat of the code, let’s see how it looks like:(Update: don’t sweat trying to find not-kept areas in the graphs: there are none)

Everything should be quite clear after the discussion above, so I’m not getting into the details here. If you’re interested in all of the graphs, you’ll find them at the end of this post.

Putting it all together
The smarter way to put this stuff all together and automate it smartly is, in my humble opinion, a makefile. Not only make will run all the actions in the right order: it will not run the whole bunch if it’s not necessary. Let’s see.

The first three lines are just variable assignments. The good thing about them is that you can override them on the command line. So if, for example, you want to test these script on a local copy of the log, you’d just use a command line like:

make LOG=/tmp/promise_summary.log

The target “all” is the first one, hence the one that will be executed if you run make alone. It will require all the images listed in the IMAGES variable to be up to date. In turn, and that’s the second target, the images will be considered up to date only when their prerequisites are up to date, including the RRD file. This brings us to the third target, the RRD file, that will be considered up to date only if the log file wasn’t updated earlier than the RRD file. If that is the case, make will run the commands to rebuild that target file; you know those command because we’ve talked about them above.

Rebuilding the RRD file starts a chain reaction, where now the RRD has been updated later than the images, so the images are rebuilt as well, and again: you know what the “create_promise_graph.sh” script does. And that would be… all.

Follow me on Twitter

Meta

Disclaimer

The postings on this site are my own and don't necessarily represent my employer's positions, strategies or opinions!!!

The original contents of this blog (posts, photos,... anything created by the author) are (C) Copyright Marco Marongiu, 2010-2018. Unless a different license is specified in the entry's body, the following license applies: "Verbatim copying and distribution of this entire article is permitted in any medium without royalty provided this notice is preserved and appropriate attribution information (author, original site, original URL) is included".