We encounter these kind of errors when the JVM calls the malloc() to get a memory buffer but there is insufficient memory, or by any native JNI code that is running under that Java process. This includes application JNI code, and third party native libraries; for example, JDBC drivers.

Could you check if you are using the native libraries that could be causing the issue.

You can also enable the -verbose:gc parameter in the server startup script to check the memory allocation pattern.

Additionally you can troubleshoot the Out Of Memory issue using any profiling tools like JProfiler, YourKit Java Profiler etc.