Metadata is key

When shipping logs from containers infrastructure it’s important to include context metadata to ensure we can correlate logs later. This becomes especially important for the Kubernetes case. You may want to watch logs from a full deployment, a namespace, pods with a specific label, or just a single container. Metadata is key to ensure you can filter logs to focus on what’s important to you.

Metadata is also useful to correlate events from different sources. When troubleshooting an issue it’s very common to check logs and metrics together, thanks to Kubernetes metadata we can filter both at the same time.

Add Kubernetes metadata

We use processors across all Beats to modify events before sending them to Elasticsearch, some of them are used to add metadata, as part of the 6.0.0 release we added add_kubernetes_metadata to the list!

add_kubernetes_metadata enriches logs with metadata from the source container, it adds pod name, container name, and image, Kubernetes labels and, optionally, annotations. It works by watching Kubernetes API for pod events to build a local cache of running containers. When a new log line is read, it gets enriched with metadata from the local cache.

Deployment

Shipping logs from Kubernetes with Filebeat is pretty straightforward, we provide documentation and sample manifests to do it. Filebeat is deployed as a DaemonSet, this ensures one agent is running on every Kubernetes node. Docker logs folder from the host is mounted in the Filebeat container, Filebeat tails all container logs and enriches them using add_kubernetes_metadata. To deploy and see it yourself, just follow these simple steps: