New "Cache Area" Heap in JVM Heap

Today’s web applications have to cater to a wider audience, they have to serve richer pages and they have to perform better than they did in the early days.One of the highly recommended techniques for improving performance is to cache as much data as possible.And it is possible to allocate large memory to processes running in the current machines (read 64 bit operating systems on 64 bit CPUs). But the one road block that holds technicians from providing a very high heap memory to Java is Garbage Collection.Java introduced the concept of Garbage Collection to reduce the pains of developers of managing memory and does a fairly good job. But it fails to deliver when it has to deal with large memory cleanups. Every JVM that exists today pauses the applications one time or the other to clean up the garbage that has been generated and during these pauses mean the response to the requests of the users are slowed down drastically. This is an unacceptable situation for most applications.At a high level there are two ways the heap is treated1. The whole heap is treated as one and memory is allocated and freed as required.2. The heap is split into multiple regions based on the age of the object. Objects are allocated in the young region and are promoted to the tenured (Old generation) region as their life increases. This is also called the generational heap and the three major JVMs today prefer this mechanism to manage the heap with some variations. Coincidentally the .Net CLR also uses a similar mechanism to manage the heap.

There are different types of collectors that exist1. Serial – This is the one with the most pause time2. Parallel – This reduces the pause times based on the number of CPUs available3. Concurrent Mark and Sweep – This further reduces the pauses as it tries to do some work during the free cycles of the application threads and starts working before the memory gets exhausted.

Most JVMs and most collection mechanisms do well in collecting the memory in the young generation regions. The problem starts when they have to collect data for the tenured generation as the objects are typically inextricably linked by the time it comes to this generation. Also this generation is of the biggest size and it contains more number of objects than the other generations. This means that it takes longer to trace the roots of the objects in this generation.

Consider the scenario in which the application caches data. This would typically get allocated in the your generation and would slowly progress to the tenured generation. If the right objects are cached then these objects will not change often. At the same time there will be other objects which will come to the tenured generation and these may not be of this nature. Now when a tenured generation collection is triggered the JVM not only has to walk through the ones that will potentially die, but it also has to go through the objects which are very unlikely to die. It is possible that the ones that will not die will occupy a larger space and will be more in number than the ones that are likely to die. This means that most of the work done in the Garbage Collection Cycle is redundant.

Now instead of this if the JVM provided a separate memory section for caching then what will happen is that the tenured will be only those objects that have stayed long for some right or wrong reason. And in a well written program these will be very few. The collection objects in this tenured area should be faster as now the objects in cache will be excluded from this Garbage Collection Cycle and this should as per our assumption be the biggest contributor to the objects in the tenured area.

The added advantage will be that the objects identified for caching will directly go to the cache region and will never trouble the young generation collection or the tenured generation collection. To handle references of the cache objects in the other areas we could maintain a reference counting algorithm which can decrement the reference counts at the time of Garbage Collection Cycles in those areas. This will ensure that one does not have to walk the young and tenured regions when running a Garbage Collection cycle in the tenured region. It will be better if we do not allow reference of these objects to be maintained by objects allocated in the other regions other than within a method.

The memory management of this area can be further enhanced by dividing this section into the following subareas:1. Area with objects that never die2. Area with objects that can be removed using LRU algorithm3. Area with objects that can be removed using LFU algorithm

With ability to tune the sizes of these regions.

Now with this it will be possible for the application developers to allocate and manage a huge cache within a single JVM which is difficult today given the limitations of Garbage Collection Cycle.