Our initial investigation revealed that this failure only happened on our build slaves with SuSE enterprise Linux 12 SP2 for every build of Elasticsearch. Other builds on these build slaves, such as our Lucene builds, were unaffected.

To reproduce the issue we started a Gradle build right from the command line on the affected machine, and it built just fine. Now the question stands: Why is the JVM unable to create a new OS thread? Do we - for some reason - create an excessive amount of threads during the build? Let’s add a test to our test suite that just spawns threads until the JVM dies:

When we ran it via Gradle on the affected system, the test had created roughly 20.000 threads when the JVM finally died. However, the number of threads reported by our “professional” thread monitoring solution (watch 'for pid in `jps -q` ; do echo -n "$pid " && ps huH p $pid | wc -l ; done' ;) ) while Jenkins was running never exceeded a few hundred threads.

Time to open Pandora’s box and to find out under which conditions the JVM throws this error. The C++ implementation of the native method java.lang.Thread.start0 is the function JVM_StartThread in jvm.cpp. Towards the end of this function, we can see that an OutOfMemoryError is thrown when the JVM was unable to create a new native thread.

We’ll leave out the details, but a native thread on Linux will be created by os::create_thread. After analysis, we came up with two error conditions:

Upon native thread creation, the JVM checks on some platforms (one of them being SuSE Linux) whether the memory addresses of stack and heap are below a certain margin, called the thread safety margin. If this safety check fails, the JVM will terminate with an OutOfMemoryError. It can be disabled by setting -XX:ThreadSafetyMargin=0.

The native thread is ultimately created by pthread_create. Whenever pthread_create returns an error code, the JVM will raise an OutOfMemoryError.

As we were just investigating and needed to reduce the failure paths to check, we started by disabling the thread safety margin check by adding -XX:ThreadSafetyMargin=0 to the JVM options of our build. Needless to say, it was not that easy and disabling the check did not change anything.

So we needed to analyze under which conditions pthread_create can fail. After reading the man page of pthread_create, we concluded that we must hit a system limit and checked:

ulimit -a which reported a limit of 64140 maximum user processes.

/proc/sys/kernel/pid_max which reported 32768. This means that the highest PID on the system can be 32768. After that PIDs wrap. Given that ulimit allows 64140 processes per user, this number was too low, and we increased it temporarily to 131072.

/proc/sys/kernel/threads-max which reported 128280. As this is above the limit of maximum user processes, we kept the value.

All in all, these numbers seemed fine. But the build was still failing. Time to go a level deeper.

pthread_create in glibc ultimately uses the system call clone. Poking through the kernel documentation, we found an interesting feature, namely the cgroup process number controller, which allows to limit the number of processes in a cgroup. This led us to look at our process hierarchy. PID 932 is the Jenkins slave process: