Brewing in Beats: disk IO status

Welcome to Brewing in Beats! With this series, we're keeping you up to date with all that's new in Beats, from the details of work in progress pull requests to releases and learning resources.

Last week we mostly worked on improving Metricbeat and reorganizing the configuration files of all our Beats. Here are the most interesting pull requests that were merged in master:

Monitor disk IO with Metricbeat

Adding disk IO information was a very requested feature in Topbeat and now is finally available in the system module of Metricbeat. Currently it works on Linux, Windows, and FreeBSD (amd64).

Standardize the exported fields

We decided to use the same format of the exported fields for all our Beats and introduce the dot notation in Metricbeat `moduleName.metricSetName` (the dot is actually a sub-document) instead of `moduleName-metricSetName`.

We are planning also to standardize the metric names and follow the rules:

use all lower case

connect multiple words by “_” (e.g. used_cpu_sys_children)

Use namespaces to group fields with “.” (e.g. connections.async.closing, connections.async.keep_alive)

Refactor the configuration files

The Beats are now following the format of the Elasticsearch yml configuration file in using the dot notation that makes the configuration file more robust as it doesn’t rely so much on indentation.

The change affects all the configuration files, but it doesn’t break compatibility as the old way of writing is still accepted.

The configuration files are containing all the options that can be configured, so they are bit too long. We decided to add a short version of the configuration file for each Beat. The short version would become the default configuration file and the long version would serve as a guide for more configuration options.

Other Metricbeat improvements

Proper shutdown in Filebeat

Clean shutdown for all Beats is an important prerequisite for features like configuration reloading and running multiple beats in a single process. It is also important for our test suite. This PR refactors the Filebeat code to make sure it shutdowns correctly.

Deprecate GeoIP support

One of the older Packetbeat features is that you can visualize the location of your clients on a map in Kibana based on the GeoIP information of the source IPs of the incoming HTTP requests.

While this is a very appreciated feature, doing GeoIP lookups in Packetbeat is not needed, because you can better do it using Logstash or the Ingest Node GeoIP plugin in Elasticsearch (new in 5.0). So, we deprecated the GeoIP in 5.0.

Switch to Perl like regular expressions

We found some limitations in the current use of our regular expression engine (guide) due to using the POSIX subset only, so we decided to move closer in 5.0 to support same general regular expression syntax used by Perl, Python and other languages.