After upgrading a cluster from 2.3 to 5.5, I'm noticing an increase in total memory. I don't have enough data yet, but it also looks to grow (slowly) unbounded until OOM. I've read other accounts (not strictly elasticsearch related) how this behavior can look like a memory leak but really it's more like an optimization ("PSST! Don't really free this memory because I'm going to need it right back, ok?").

Elasticsearch is running inside a container. It has 30G heap. The host system has 128G. Using openjdk 1.8.0_131.

1st question, how can I restrict the amount of off-heap usage in order to avoid OOM's? I tried setting -XX:MaxDirectMemorySize and also tried MALLOC_ARENA_MAX=2. The malloc arena experiment didn't yield much different behavior: total resident memory quickly grows 80+G. Setting MaxDirectMemorySize=32G improved the situation slightly: resident memory reached around 64G but still continued to climb, albeit at a much slower rate.

2nd question, how can I better understand what is driving the off-heap usage? An increase in this usage from 2.x to 5.x was expected since doc_values are the default. Any tips or references are greatly appreciated. Thanks!

I'm looking for ways to identify or instrument the parts of elasticsearch that are responsible for off-heap usage. I have a shallow understanding that it's mostly from memory mapped byte buffers. My expectation was that -XX:MaxDirectMemorySize would limit that but it's not.

I'm pretty sure it's not from thread stack size. We do see more threads in v5 than v2, but not enough to account for the extra usage. Looking at MetaSpace size now. Curious if anyone has insight into what in elasticsearch could be driving that. Dynamic classes?

When stored fields are configured with BEST_COMPRESSION, we rely on garbage collection to reclaim Deflater/Inflater instances. However these classes use little JVM memory but may use significant native memory, so if may happen that the OS runs out of native memory before the JVM collects these unreachable Deflater/Inflater instances. We should look into reclaiming native memory more aggressively.