We are happy to announce the release of Hazelcast Jet 0.6! This version brings many new improvements which I will try to squeeze into this blog post.

Improved Streaming Support for the Pipeline API

With 0.5, Jet introduced a new high-level API called Pipelines which offered a convenient way to do distributed data processing using a powerful and expressive DSL. With 0.6, this API is now overhauled and extended to support a more complete set of features, including full streaming support with windowing. Previously, stateful streaming operations such as windowing were only possible to do using the core API. An example of a streaming job which uses windowing and uses a map event journal as the source is given below:

Stateful Transforms With the Pipeline API

Another improvement to the Pipeline API is the ability to stateful map, flat map, and filter operations. This is useful when the stage involves some external library or object that needs to be initialized and then re-used for each item. One example of this is using a Hazelcast Replicated Map for enriching a stream:

Merging and Updating Map Sinks

Hazelcast Maps have a feature called entry processor which allows entries in the map to be updated in place in a thread-safe manner. Previously, the map sink in Jet would always overwrite the existing entry. With 0.6, we are introducing merging and updating sinks, which allow the sink to update the entry in place instead.

Job Management Improvements

In Jet 0.5, there were big changes to how job submission works and it was the first release to have fault-tolerance with jobs restarting automatically upon a node failure. With 0.6, we have improved how job cancellation works and added the ability to name jobs.

Manual Job Scale Up

It has always been possible to add new Jet members to a running cluster without affecting the running jobs, however, it was not possible to make use of the new computational capacity added by the new members. Now jobs can be restarted manually using the job.restart() method. Once a job is restarted, it will run on the whole cluster including the new members, and if processing guarantee for the job is set to at-least-once or exactly-once, it will use the latest available snapshot to restore the job state.

Improved Java 9 Support

The main Jet code base now builds on Java 9 without any problems. We have also added Automatic Module Names to all the modules so that Jet can be used as a module in Java 9 projects.