Reliable Insights

Prometheus 2.9.0 is now out, following on from 2.8.0 with many fixes and improvements.

Issues around missing time series when remote write reading from the WAL, which was introduced in 2.8.0, have been resolved. This should also fix the "unknown series references" warning at startup. A clean WAL (or waiting for all time series to churn out) is required for the issue to be fully gone. Remote write is also more memory efficient, the CPU impact of flaky targets has been reduced, and for the third release in a row there's improvements to indexing in the TSDB.

There's a honor_timestamps option to deal with misbehaving targets producing unnecessary and problematic timestamps. TLS certs are now automatically reloaded from disk, and scrapes are now staggered across Prometheus servers (previously they were only staggered within a Prometheus server). In service discovery OpenStack and Kubernetes have more metadata, and Consul now supports multiple tags if you're using the tag (now tags) config field for performance.

There's other small improvements too, including slightly more readable log output, the status page has more information, and additional endpoints work via POST.

There’s also a number of improvements, bug fixes and other changes, see the release notes for more detail.