Working with MongoDB TTL Indexes from Java

In the previous blog post we have seen how to leverage the ability of MongoDB to automatically delete documents based on the size with Capped Collections.

Another common use case is to delete documents based on an expiry date. For instance log events that you only want to store for certain days or session data that should only be stored for a few hours. MongoDB supports this use case, not with a special collection, but with a special index: Time to live index

You can create a TTL index on any date field or any date array field. When you create the index you specify the amount of time after the document should expire.

You can use a TTL index for queries like a normal index. When you index a date array with a TTL index MongoDB chooses the earliest (lowest) date in the array for computing the expiry date. MongoDB automatically deletes documents after the indexed date + specified amount of time has passed. If the indexed field in the document is missing or not a date or not a date array the document will never expire.

MongoDB runs a background thread every 60 seconds that checks the TTL indexes and deletes the documents. Therefore, a TTL index does not guarantee that a document is immediately deleted when the expiry date is reached. Most of the time there is a delay between expiration date and the time MongoDB deletes the document.

You can also insert documents with an expiry date in the past. Such a document will not be deleted immediately, MongoDB removes it the next time the background thread runs.

The previous example expired all the documents after date + 1 minute. But what if we want to expire the documents with different expiration dates. For instance, we store log events and we want to expire logs with the severity INFO after 1 hour, with the severity WARN after 7 days and ERROR logs after 1 month.

For that use case we add a field to our document that specifies the expiration date and create a TTL index on this field with a expireAfter of 0.

MongoDB deletes the documents at the specified date and time. As usual there will a little delay between the specified expiration date and MongoDB deleting the document, because the background thread only runs every 60 seconds