Today I found myself in a need of a least-recently-used (LRU) cache system. Essentially, I wanted a cache of the last 10 000 used objects. When a new object was to be loaded, I wanted to unload the oldest element in the cache. Before, the cache was only 96 elements, at which point looping through the list to find the least recently used element was perfectly fine for the load I had, but with 10 000 objects, that situation changed. I found that LinkedHashMap could be used as a makeshift cache, as it can be set to update the internal order of elements when they are accessed, but it just seemed inefficient and weird for what I actually wanted to accomplish here. Some research made me realize that a simple linked list was a good fit for this, as a double linked list can easily be maintained in LRU order. Basically, whenever an element in the cache is used, it is moved to the beginning of the list (an O(1) operation on linked list), meaning that the cache stores objects sorted from most recently used to least recently used. Then to remove the most least recently used object, I just remove the tail of the list, again an O(1) operation.

Now, I hope we all know how bad Java's LinkedList class is. Not only does it generate garbage every time an object is added to it, but moving an object to the beginning of a list is an O(n) operation as I'd need to use list.remove(object), which would need to scan through the list until it finds the object in question and can remove it. A proper linked list implementation would store the previous and next references in the object itself, meaning that finding the previous and next objects become O(1) operations, and the element can be removed without going through the entire list. So... I wrote a minimal linked list that could do what I needed and not much more.

As mentioned, the main difference is that my linked list class does not allocate Node objects to store the next and previous references, but instead stores those references in the elements themselves. This leads to a couple of peculiarities.

- All elements must implement an interface that allows the list to set the previous and next pointers (or alternatively extend a class which implements those functions for it). - You can't place the same element in a linked list twice, as each element can only track one previous and next neighbor. - For the same reason, you can't even use the same element in two DIFFERENT lists at the same time.

In my case, none of these were an issue, so I went ahead and implemented it. I even added a special method for moving an element in the list to the start of the list for maximum performance in that sense. The class provides the following functions, all of which are O(1):

There is no random access function. The correct way of looping through the list is to simply call element.getNext() until it returns null (the current size of the list is tracked though). The somewhat complicated usage of generics is there to allow for type safety, both when extending the Element class and when working with FastLinkedList.

Remove remaining 98 000 objects using removeFirst(): LinkedList: 0.298 ms FastLinkedList: 2.78 msNoticeably slower in this test due to worse cache coherency. FastLinkedList needs to access each element's data to find the next and previous node in the list. As the elements are shuffled, this ends up jumping around randomly in memory. Java's LinkedList instead creates Node objects for each element, which end up sequentially in memory as they're recreated each iteration. This is not an issue with real-life data, and is the cost of going garbage-less.

The tail pointer would point at the index at which the least used object (last added) object is.The head pointer would point at the next available spot (or a taken spot if it has looped all the way around)

When you remove an element, simply set it to null in the array (make sure all operations have a null check)To move an element from index N to the front, just save it to a temp var, set index N to null, then put the saved element at the head pointer and increment the head pointer.

Instead of the Element interface containing next and previous pointers, it could contain an index into the array so that lookups are O(1)

The problem with a simple ring buffer is that it's essentially a FIFO queue. When an element is accessed, I need to move it to the start of the queue again to make sure it's the most recently used again. I don't think that's doable in an array-based ring buffer. Removing an element would leave a null hole in the array. If you want to fill the hole, you need to shift half the elements to fill the hole (either the ones before or after, so 0 to size/2 elements). If you just leave the hole there, you've essentially reduced the size of the cache until the head reaches that point and can fill the null hole, right? I imagine a pathological (but very common) case where the cache is full of objects, and then two elements are accessed repeatedly, causing them to be moved to the head of the list repeatedly. If each movement leaves a null hole, then one element would be evicted from the cache each time an element is moved to the front of the list. =/ If I've misunderstood or missed something, please let me know.

Nice! Are you thinking about a concurrency-safe version by chance, or is that definitely not an issue in your situation?

Multi-threading is on my mind as I just used a LinkedBlockingDeque for safely managing concurrent calls to an AudioCue class that allows concurrent playback. Am only using offerFirst and pollLast methods. Only a handful of instances per Cue play at one time, so efficiency is less of a concern than reliability.

Are you thinking about a concurrency-safe version by chance, or is that definitely not an issue in your situation?

Not really, it's not a requirement for my use case. It'd just get messy if I needed to support concurrency in the system I'm working on, really. You can always just synchronize on the whole class if you really wanted to.

I'd be inclined to have a reference in the Element, to the List that owns it.

It'd allow fail-fast protection against multiple logic errors (adding an Element to multiple lists/the same list multiple times, removing/moving an element from a List that it's not a member of, etc), making the code far less fragile.

imagine a pathological (but very common) case where the cache is full of objects, and then two elements are accessed repeatedly, causing them to be moved to the head of the list repeatedly. If each movement leaves a null hole, then one element would be evicted from the cache each time an element is moved to the front of the list.

Actually you could make use of this. Just don't move the object to the top of the list, if it's in the upper half (or some other threshold) already. If anything, just swap it with an object a couple of indicies up in the buffer. This way you would only produce holes for the less used objects. Since you don't need additional references, you can easily make up for the lost cache entries by increasing the array size.

Or you only move the head pointer for adding new objects and just swap existing objects up in the buffer. This would not produce any holes and while it is technically not an LRU list anymore, it might provide good enough heuristics to function as a cache.

I like your solution, esp. because its an intrusive list - the node pointers are members of the list element.

NOTHING in this list checks for errors. I should've mentioned that in the original post I guess. xd

>Add already existing element again ---> Corrupt link references.>Add element from another list ---> Corrupt other list.>moveToHead() on object not in the list ---> head is set to object not in the list.>remove() object not in list ---> corrupts size counter and may corrupt link references.>removeFirst/Last() on empty list ---> nuppo

I keep track of if an element is in the list outside the list for the cache, so such testing wasn't necessary (and IMO not the job of the list in the first place).

I'd be inclined to have a reference in the Element, to the List that owns it.

It'd allow fail-fast protection against multiple logic errors (adding an Element to multiple lists/the same list multiple times, removing/moving an element from a List that it's not a member of, etc), making the code far less fragile.

Yeah, that'd be how you add testing. Add a reference to the list the element currently is on, check this reference in add*(), moveToFirst() and remove(). Null out the reference on removal. Also add an isEmpty() check for the removeFirst/Last() functions.

Actually you could make use of this. Just don't move the object to the top of the list, if it's in the upper half (or some other threshold) already. If anything, just swap it with an object a couple of indicies up in the buffer. This way you would only produce holes for the less used objects. Since you don't need additional references, you can easily make up for the lost cache entries by increasing the array size.

Or you only move the head pointer for adding new objects and just swap existing objects up in the buffer. This would not produce any holes and while it is technically not an LRU list anymore, it might provide good enough heuristics to function as a cache.

I'm worried about experimenting too much with my use case. The cache is critical for performance, and if I were to start thrashing, the game will essentially freeze up. I could try some kind of system where you don't move an accessed element to the head of the list, but just "upgrade" it a couple of elements up by swapping it with an element x indices up in the list. If x is randomly chosen for each swap, then the risk of getting to a pathological case would be low. Like you said, this wouldn't be an LRU cache anymore, but some kind of heuristic priority cache I guess. It's an interesting idea, but I'm afraid I don't have much more time to spend on this right now. =/

>Add already existing element again ---> Corrupt link references.>Add element from another list ---> Corrupt other list.>moveToHead() on object not in the list ---> head is set to object not in the list.>remove() object not in list ---> corrupts size counter and may corrupt link references.

Instead of exposing the internal element class, won't it be better to keep a static pool of element objects in the linked list class? That's better for the API and easily fixes the corruption problems. I'd like to see how that performs over the standard list.

Does your LRU cache need to be perfect? If not, I'd just create buckets of items, and transfer items among buckets, which is cache friendly.

List<List<Item>>

where the inner list is an ArrayList, and the outer list... probably too (as that is still bound to be faster than the LinkedList, even on random insert/remove) given that you probably won't have more than 100 buckets for 10.000 items.

Hi, appreciate more people! Σ ♥ = ¾Learn how to award medals... and work your way up the social rankings!

I'd just create buckets of items, and transfer items among buckets, which is cache friendly.

Its good concept – but its not so easy to implement as basic ListAnd about LRU cacheIt sad, but it not help much – because Java hold pointer to objectAnd objects scattered all around random places on memorySo on iteration of array objects – it jumps around memory - its not cache friendly)

@SHCA pooled Node version would probably be slightly worse performance-wise when looping over the list due to the Node object and the element object being in two different places in memory. Also, without exposing the pointers, it becomes a bit annoying to loop through the list, but that's no real concern I guess.

@jonoInteresting, it'd be cool to see a.comparison between that and my implementation.

@jonoInteresting, it'd be cool to see a.comparison between that and my implementation.

To make it comparable it has to be turned into a doubly-linked list and be generalised to hold objects instead of just ints.

I tried this out, using elements with a "getID" method to enable the random access. It loses the cache coherency from storing data+references in one array, as well as having more overhead from the extra book-keeping of the previous links. In the end approximately 500x faster than java.util.LinkedList on the random access operations but roughly the same on the removeFirsts -- there is a lot of variance between tests on this though.

In short, no good reason to move from your implementation. It moves the next/prev links out of the elements and into the single pool class, but at the cost of some efficiency.

java-gaming.org is not responsible for the content posted by its members, including references to external websites,
and other references that may or may not have a relation with our primarily
gaming and game production oriented community.
inquiries and complaints can be sent via email to the info‑account of the
company managing the website of java‑gaming.org