Brewing in Beats: Enrich events with Kubernetes metadata

Welcome to Brewing in Beats! With this weekly series, we're keeping you up to date with what's new in Beats, including the latest commits and releases.

Enrich each event with Kubernetes metadata

Thanks to Vijay Samuel, there is a community contribution to add support for Kubernetes as a processor for Beats. The Kubernetes processor allows Beats to enrich events with metadata coming from the Kubernetes Pod from which the event originated. By default, the Kubernetes processor adds the following metadata:

pod name

pod namespace

container name and labels

Depending on the Beat, the Kubernetes processor can add additional information. For example for Filebeat, it takes the source fields, extracts the container ID, and uses it to retrieve metadata about the Pod from which the log message originated.

This will be released as Beta in 6.0.

Monitor Elasticsearch with Metricbeat

A new module is added to Metricbeat for monitoring Elasticsearch. It exports metrics about:

Node info (node metricset) by interrogating the Cluster API of Elasticsearch to get cluster nodes information. This metricset only fetches the data from the _local node, so it must run on each Elasticsearch node.

Node stats (node_stats metricset) by calling Cluster API of Elasticsearch to get the cluster nodes statistics. This metricset only fetches the data from the _local node, so it must run on each Elasticsearch node.

The Elasticsearch module in Metricbeat exports only a few basic metrics, and for a better monitoring experience, we recommend you to use X-Pack monitoring.

This will be released as Beta in 6.0.

New community Beat: Kafkabeat

Kafkabeat is built on top of libbeat infrastructure to read the streaming events stored in Kafka and send them to Elasticsearch. An option would be to use Kafkabeat in the following scenario: Filebeat >> Kafka >> Kafkabeat >> Elasticsearch.