Elasticsearch 1.4.1 and 1.3.6 Released

Today, we are happy to announce the bug fix release of Elasticsearch 1.4.1, based on Lucene 4.10.2, and the bug fix release of Elasticsearch 1.3.6. You can download them and read the full changes list here:

Shard allocation

In Elasticsearch 1.3.0, we turned on disk based shard allocation by default. If a node's disk usage crosses the low watermark — 85% — no more shards will be allocated to it, and once it crosses the high watermark — 90% — shards are actively moved to other nodes.

Elasticsearch 1.4.1 adds three improvements to disk based shard allocation:

Disk usage was only checked when a shard somewhere in the cluster needed to be allocated. Now, disk usage is checked every 60 seconds. (#8270).

Disk full messages were only logged at the DEBUG level, which made it hard to figure out why new shards were not being allocated. Now, warnings are logged at the WARN level every 30 seconds. (#8382).

Previously, the allocation decider only took into account the size of shards currently on the node when deciding whether to move a shard to another node or not. Now, it also takes into account shards that are in the process of being moved away. This reduces the amount of shard movement to the minimum required. (#8569).

Parent/child and nested documents

Elasticsearch 1.4.0 built and cached fixed bitset filters for parent/child and nested documents eagerly — when opening a new segment — to ensure that queries, filters, and aggregations are always fast. A few users with many nested fields experienced much higher heap usage than with earlier versions.

By changing the order in which documents are processed by a nested aggregation (#8454), fixed bitset filters are no longer needed for child documents. Now, we only cache a filter representing parent docs, that is the docs which are not nested. This greatly reduces the size of the cache space needed. (#8414, #8440).

Date ranges

Two long standing issues with date ranges have been fixed in this release. The first is to do with how dates are rounded. For instance, assume that the timestamp field contains timestamps with a resolution of 1 second. A range filter with { "lt": "2014/11/26||/d"} should find all timestamps less than 2014/11/26 00:00:00. However, if we change lt to lte, then we also want to include any timestamps up to but not including 2014/11/27 00:00:00.

Previously, lte would have incorrectly included timestamps of 2014/11/27 00:00:00. Instead, we now subtract one millisecond from the upper range so that lte works as expected. (#8556).

The second bug involved alias or percolator filters that used now() in a date range clause. The value for now() was determined at the moment that the filter was created, instead of being updated whenever the filter was executed. With #8534, now() works as you would expect for filters in both aliases and percolators.