Managing the Off-heap Cache

Each Replication Node in the store divides the memory available
to it between memory containing store objects, memory used by
the host operation system (which includes the file system
cache), and the in-memory cache used to contain frequently
accessed store data. The amount of memory used by the cache is
configurable using the rnHeapPercent
parameter. This is expressed as a percentage of the total
memory available to the Replication Node (which is, in turn,
determined by the amount of memory made available to the Java
VM). By default, 70% of the memory available to the Replication
Node is reserved for the in-memory cache.

However, this can and probably will fill up. When that
happens, objects are evicted from the cache based on a
Least-Recently-Used (LRU) algorithm which guarantees that the
most commonly accessed objects will remain in the cache.

It is possible to configure your Replication Node so that the
in-memory cache is very large, and therefore is less likely to
overflow. However, several drawbacks occur when the cache
becomes too large. The most important of these is that Java
Garbage Collection performance can decrease, which will have an
adverse impact on your Replication Node's overall performance.
Therefore, it is in general a good idea to keep your
heap size below 32 GB. Doing so limits your
in-memory cache size to about 22.4 GB.

Note

Be aware that Oracle NoSQL Database limits your maximum heap size to a
default value of 32 GB so as to prevent accidentally
creating an excessively large cache. This value is managed
using the rnHeapMaxMB, which is described in
greater detail later in this section.

When your Replication Node's heap is sized lower, it is
possible to configure the system such that any memory unused by
the heap will be used for the operating system and file system
cache. A large file system cache has performance benefits, but
it also has drawbacks:

There is a significant redundancy between the main
cache and the file system cache because all data and
Btree information that is logged (written) by the
Replication Node appears in the file system and may
also appear in the main cache.

It is not possible for dirty Btree information to be
placed in the file system cache without logging it.
This logging may be otherwise unnecessary, and the
logging creates additional work for internal cleaner
threads.

To avoid the problems caused by excessively large in-memory and
file system caches, your Replication Node is by default
configured to use an off-heap cache. The
off-heap cache is used to hold record data and Btree nodes when
these are evicted from the "main cache" because it overflows.
When the off-heap cache overflows, eviction occurs according to
the same LRU algorithm used by the main cache.

Note

You should engage in performance testing before
modifying the caches in use by your production store.

Configuring the Off-heap Cache

You do not directly control the amount of memory available
to the off-heap cache. Instead, you make memory available
to it mainly by limiting the amount of memory available to
the operating system. You might also need to control the
size of the Java heap, which in turn controls the size of
the in-memory cache. Any Replication Node memory that is
left over once the heap and operating system requirements
are met is used for the off-heap cache. If no memory is
left over, the off-heap cache is not used.

The parameters you use to (indirectly) configure the off-heap cache
are:

systemPercent

This defines the percentage of memory available to
the Storage Node that is available for operating
system use after the heap
requirement is subtracted. By default, this value
is 10%. Configuring this value to a number that is
greater than 10% but less than 100% might leave
room for the off-heap cache (depending on the
memory on your Storage Node and the value chosen
for this parameter). If room is available for the
off-heap cache, it will be turned on.

For most production systems, if you are using the
off-heap cache, then the default value of 10% for
this parameter should be sufficient.

rnHeapMaxMB

This is the maximum amount of memory available for
the Java heap. (The heap is where the
in-memory cache is contained.)
The size of your heap will be the lesser of this number,
or the size arrived at using the
rnHeapPercent parameter value:

total SN memory * rnHeapPercent = Heap Size

rnHeapPercent is by default 85%.
So if your Storage Node has 32GB memory, then
the heap size will be 32 * 0.85 = 27.2 GB. However,
if you set your rnHeapMaxMB to
25,600 (25*1024), then
your heap size will be 25GB instead.

Remember that the heap size is not the same as the
in-memory cache size. The in-memory cache size is
expressed as a percentage of the heap size. By
default, this is 70% (configurable using
the rnCachePercent parameter).
So if your heap size is 27.2 GB, then your
in-memory cache size ends up being 19.04 GB (27.2 *
0.7).

Note

Regardless of the values provided to the
parameters described here, and the actual
memory available for the heap, your heap size
is limited to a maximum size of 32 GB.

For example, if your Storage Node has 64GB of memory, and
just one Replication Node, then by default:

The heap size is 32GB. (64 * .85 = 54.4 which is
greater than the 32GB maximum.)

The in-memory cache size is 22.4 GB (32 * 0.7).

A system memory size of 6.4 GB. (64 *.1)

An off-heap cache size of 25.6 GB. This is the
amount of memory left over after the heap and
system requirements have been met.

If you want to eliminate the off-heap cache, then set
system memory to 100%. Do this for each Storage Node in
your store. Use the following commands: