JDO : Troubleshooting

This section describes the most common problems found when using DataNucleus in different architectures.
It describes symptoms and methods for collecting data for troubleshooting thus reducing time to narrow
the problem down and come to a solution.

Out Of Memory error

Introdution

Java allocate objects in the runtime memory data area called heap. The heap is created on
virtual machine start-up. The memory allocated to objects are reclaimed by Garbage Collectors when
the object is no longer referenced
(See Object References).
The heap may be of a fixed size, but can also be expanded when more memory is needed or contracted
when no longer needed. If a larger heap is needed and it cannot be allocated an OutOfMemory
is thrown. See
JVM Specification.

Native memory is used by the JVM to perform its operations like creation of threads, sockets,
jdbc drivers using native code, libraries using native code, etc.

The maximum size of heap memory is determined by the -Xmx on the java command line. If Xmx is not
set, then the JVM decides for the maximum heap. The heap and native memory are limited to the
maximum memory allocated by the JVM. For example, if the JVM Xmx is set to 1GB and currently use of
native memory is 256MB then the heap can only use 768MB.

Causes

Common causes of out of memory:

Not enough heap - The JVM needs more memory to deal with the application requirements.
Queries returning more objects than usual can be the cause.

Not enough PermGen - The JVM needs more memory to load class definitions.

Memory Leaks - The application does not close the resources, like the PersistenceManager
or Queries, and the JVM cannot reclaim the memory.

Garbage Collection - If no full garbage collection is performed before the OutOfMemory it
can indicate a bug in the JVM Garbage Collector.

Memory Fragmentation - A large object needs to be placed in the memory, but the JVM cannot
allocate a continous space to it because the memory is fragmented.

JDBC driver - a bug in the JDBC driver not flushing resources or keeping large result sets
in memory.

Throubleshooting

JVM

Collect garbage collection information by adding -verbosegc to the java command line.
The verbosegc flag will print garbage collections to System output.

Sun JVM

The Sun JVM 1.4 or upper accepts the flag -XX:+PrintGCDetails, which prints detailed
information on Garbage Collections.
The Sun JVM accepts the flag -verbose:class, which prints information about each class loaded.
This is useful to troubleshoot issues when OutOfMemory occurs due to lack of space in the PermGen,
or when NoClassDefFoundError or Linkage errors occurs.
The Sun JVM 1.5 or upper accepts the flag -XX:+HeapDumpOnOutOfMemoryError, which creates a
hprof binary file head dump in case of an OutOfMemoryError. You can analyse the heap dump using
tools such as jhat or YourKit profiler.

DataNucleus

DataNucleus keeps in cache persistent objects using weak references by default. Enable debug mode
DataNucleus.Cache category to investigate the size of the cache in DataNucleus.

Resolution

DataNucleus can be configured to reduce the number of objects in cache. DataNucleus has cache for
persistent objects, metadata, datastore metadata, fields of type Collection or Map, or query results.

Query Results Cache

The query results hold strong references to the retrieved objects. If a query returns too many
objects it can lead to OutOfMemory error. To be able to query over large result sets, change the
result set type to scroll-insensitive in the pmf setting
datanucleus.rdbms.query.resultSetType.

Query leak

The query results are kept in memory until the PersistenceManager or Query are closed. To avoid
memory leaks caused by queries in memory, it's capital to explicitly close the query as soon as
possible. The following snippet shows how to do it.

Cache for fields of Collection or Map

If collection or map fields have large number of elements, the caching of elements can be disabled
with the property datanucleus.cache.collections setting it to false.

Persistent Objects cache

The cache control of persistent objects is described in the Cache Guide

Metadata and Datastore Metadata cache

The metadata and datastore metadata caching cannot be controled by the application, because the
memory required for it is insignificant.

OutOfMemory when persisting new objects

When persistent many objects, the flush operation should be periodically invoked. This will give a
hint to DataNucleus to flush the changes to the database and release the memory. In the below sample
the pm.flush() operation is invoked on every 10,000 objects persisted.

Throubleshooting

Database locking

Use a database specific tool or database scripts to find the current database locks.
In Microsoft SQL, the stored procedured sp_lock can be used to examinate the database locks.

Query Timeout

To avoid database locking to hang the application when a query is performed, set the query timeout.
See Query Timeout.

Garbage Collection pauses

Check if the application freezes when the garbage collection starts. Add -verbosegc to the
java command line and restart the application.

Application Locking

Thread dumps are snapshots of the threads and monitors in the JVM. Thread dumps help to diagnose
applications by showing what the application is doing at a certain moment of time.
To generate Thread Dumps in MS Windows, press <ctrl><break> in the window running
the java application.
To generate Thread Dumps in Linux/Unix, execute kill -3 process_id

Command Line Tools

CreateProcess error=87

Problem

Windows has a command line length limitation, between 8K and 64K characters depending on the Windows version, that
may be triggered when running tools such as the Enhancer or the SchemaTool with too many arguments.

Solution

When running such tools from Maven or Ant, disable the fork mechanism by setting the option fork="false".