One version to rule them all

You might be wondering why we’re jumping from version 1.2 directly to 5.0. To make our software suddenly more stable and to one-up our competition, of course. In seriousness, all the projects in the Elastic stack are doing releases in sync and will use the same version numbers from now on. As Kibana is currently at 4.5, we’re all going with 5.0 as the next major.

This is to avoid the support matrix from hell, for example now you need to know that Beats 1.2 were tested against Elasticsearch 2.3, Logstash 2.3 and Kibana 4.5. Starting with 5.0, you’ll know that if Beats and Elasticsearch have the same version number, they were released at the same time and we have tested them together. It simplifies communication all around.

New Features

Beats 5.0-alpha1 comes packed with new features and you can expect more of them to land in during the alpha and beta phases. Here are some of the highlights from Alpha 1:

Custom fields and generic filtering

You now have more freedom over how the documents created by the Beats look like. On one hand, you can now add custom fields and tags per Beat and module. On the other hand, you can use the newly introduced generic filtering to remove the fields that you don’t want. These features are implemented at the libbeat level, meaning that all community Beats automatically benefit from them as soon as they upgrade.

JSON support in Filebeat

Filebeat can now natively decode JSON objects from log lines. This is useful for structured logging, where the logging library writes the metadata directly formatted as JSON. This can also be used as a convenient way of collecting logs from Docker hosts, because Docker uses JSON to wrap the log lines from the application.

Integration with Ingest Node

The new Ingest Node functionality, released with Elasticsearch 5.0.0-alpha1, is big news because it gives users processing capabilities similar with Logstash directly in Elasticsearch! This makes it really easy to get started with the Elastic stack. For simple logging usecases, for example, you only need Filebeat and Elasticsearch.

All Beats can work with the Ingest Node, simply set the pipeline parameter in the Elasticsearch output configuration.

Packetbeat IP/TCP flows

So far Packetbeat was focused on the application layer protocols, giving you visibility into the business transactions as seen in the network. Packetbeat now also reports statistics like packet count and byte count about IP and TCP flows, regardless of the upper layer protocols. This opens Packetbeat to a new set of use cases, giving insights into how the traffic is flowing through the network.

Kafka output

We listened to your feedback and we’ve added Kafka output support in Beats, at the same time removing the deprecation mark for the Redis output. This means that if you are passing all messages through a Kafka queue anyway, you won’t need a Logstash instance to convert between Beats and Kafka.

Winlogbeat improvements

Winlogbeat now extracts all the fields from Windows event log records including the EventData and UserData fields and includes them in the documents it indexes. In addition, now it is possible to select events by event ID, level, and provider. Winlogbeat efficiently implements this event selection by using a query with Windows APIs so that only the requested events are returned.

Don’t fear the alpha

Getting your feedback early is key for us to make the necessary adjustments in time for the 5.0 GA release, so please test early and often. You can find us on discuss for question and discussion and on Github for issues and enhancement requests.