Elasticsearch 6.3.0 Released

Editor's Note (September 7, 2018): This post refers to X-Pack. Starting with the 6.3 release, the X-Pack code is now open and fully integrated as features into the Elastic Stack.

Today we are pleased to announce the release of Elasticsearch 6.3.0, based on Lucene 7.3.0. This is the latest stable release, and is already available for deployment via our Elasticsearch Service on Elastic Cloud.

These are all things which we hope we can make inroads into our new Elasticsearch SQL release. Our hope is to allow developers, data scientists, and others that are familiar with the SQL language -- but so far unfamiliar with or unable to use the Elasticsearch query language -- to use the speed, scalability, and full-text power that Elasticsearch offers and others have grown to know and love.

If you’re just getting started using this functionality or the power of Elasticsearch that powers it, here are a few things to try:

SELECT … ORDER BY SCORE() DESC to be able to sort by the relevance of the search results

Get all of the full-text magic from tokenization to stemming by using the MATCH operator like SELECT … WHERE MATCH(fieldname, 'some text')

This is still the very early days of our SQL efforts, so we’re marking it as experimental while we receive feedback from you on any bugs and feature requests. In the future, we hope to release an ODBC driver as well as simplify and build out our existing JDBC driver including other functions and predicates so that we can have better compatibility with other 3rd party tools like BI tools. In the meantime, please do let us know what you’d like to see next and any problems you run into!

Note that this feature is made available in the “default” (non-OSS-only) distribution of Elasticsearch and the REST API -- including the “translate” functionality and the CLI tool are completely free.

Rollups

The Elastic Stack has been increasingly used as a datastore and visualization engine for IoT, performance monitoring, and other metrics data. And while we continue to believe that most data tends to have value over a long period of time, the relative value of the data can decrease as it ages. With the release of 6.3, we can now create a job in Elasticsearch which will periodically roll up aggregate statistics about recently added data and make it available for search.

Rollups are not themselves completely new in the metrics storage world, but we’ve done a few automatic things that we think are pretty neat:

When you ask Elasticsearch to store a rollup of data like an average, it will automatically store underlying statistics like sum and count so that it can recompute the average on at query time to help you avoid the Simpson’s paradox.

Note that this feature is made available in the “default” (non-OSS-only) distribution of Elasticsearch and is completely free.

Java 10 Support

With the previous release of Elasticsearch, we announced support for Java 9. Java 9 is a short term release, and has already reached EOL this past March 2018. To continue support of Java short term releases, we are announcing support of Java 10 in Elasticsearch 6.3. Java 10 is also a short term release, and scheduled to reach EOL in September 2018.

As previously mentioned, we strongly advise the majority of users to stick with Java 8, unless they are comfortable with the rapid release cycle of Java short term versions (and EOL dates). Our support matrix is a handy reference for JVM options for Elasticsearch.

Security Updates

This update also fixes two security issues.

Elasticsearch Information Exposure Vulnerability (ESA-2018-10)

In Elasticsearch versions 6.0.0-beta1 to 6.2.4 a disclosure flaw was found in the _snapshot API. When the access_key and security_key parameters are set using the _snapshot API they can be exposed as plain text by users able to query the _snapshot API.

Although it is advised in the 6.X _snapshot API documentation to define the access_key and security_key parameters in the keystore, it is still possible to define them outside of the keystore using the API.

Affected Versions: Elasticsearch versions 6.0.0-beta1 to 6.2.4

Elasticsearch Information Exposure Vulnerability (ESA-2018-11)

A sensitive data disclosure flaw was found in the Elasticsearch repository-azure (formerly elasticsearch-cloud-azure) plugin. When the repository-azure plugin is set to log at TRACE level Azure credentials can be inadvertently logged.

Affected Versions:All versions of Elasticsearch

Known issue with Elasticsearch 6.3.x on JDK 10

There appears to be a bug in JDK 10 impacting Elasticsearch 6.3.x (note that the official Docker images are based on JDK 10). The issue appears to impact the C2 compiler on hardware that supports AVX-512 instructions (e.g., Skylake-X CPUs). The issue manifests as a segmentation fault in Java code executing org.apache.util.MergedIterator#pullTop:

A workaround is to disable the C2 compiler from using AVX-512 instructions via the JVM option-XX:UseAVX=2. This issue has been taken upstream to the JDK maintainers and the workaround will be necessary until a version of the JDK that is not prone to this issue is released. For additional background, see issue 31425.