Version Purge is running and response times from AEM are slow. Thread dumps [6] show that version purge is running and blocking other threads.

Environment

AEM 6.0, 6.1 and 6.2

Cause

Version Purge maintenance task runs past its scheduled window of time if it takes too long. The cause of this issue is likely due to bulk version creations in AEM that caused Version Purging to take too long.

2. Modify the maintenance window

Move the version purge maintenance schedule earlier to avoid it from running during business hours. See this documentation for how to modify maintenance schedules

3. Investigate what application code is causing the versions to be created and optimize it.

If versions are being created due to activations then modify the code so it suppresses version creation.
Use the ReplicationOptions [1] class, call ReplicationOptions.setSuppressVersions(true) [2] and pass the object as a parameter when calling Replicator.replicate [3].

4. Disable Versioning on Companion app and WebDav uploads

5. Enable auto-purging on version creation

The Version Manager has a feature where it can purge old versions as you add new ones to pages and assets. Follow these steps to enable this feature.
a. Go to http://aem-host:port/system/console/configMgr/com.day.cq.wcm.core.impl.VersionManagerImpl
b. Enable "Enable Purging"
​c. Save

6. Run Offline Tar Compaction

Make sure you are running offline tar compaction on a regular basis as a lack of compaction maintenance can affect system performance overall.

pool-20-thread-17 - priority:5 - threadId:0x00007fbc5c00c800 - nativeId:0x341b - state:RUNNABLE
stackTrace:
java.lang.Thread.State: RUNNABLE
at java.io.RandomAccessFile.readBytes(Native Method)
at java.io.RandomAccessFile.read(RandomAccessFile.java:377)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:436)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:416)
at org.apache.jackrabbit.oak.plugins.segment.file.TarWriter.readEntry(TarWriter.java:186)
- locked <0x0000000679fe3910> (a org.apache.jackrabbit.oak.plugins.segment.file.TarWriter)
at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.readSegment(FileStore.java:847)
- locked <0x00000005c0c49560> (a org.apache.jackrabbit.oak.plugins.segment.file.FileStore)
at org.apache.jackrabbit.oak.plugins.segment.SegmentTracker.getSegment(SegmentTracker.java:182)
at org.apache.jackrabbit.oak.plugins.segment.SegmentId.getSegment(SegmentId.java:108)
- locked <0x00000004dcafb448> (a org.apache.jackrabbit.oak.plugins.segment.SegmentId)
at org.apache.jackrabbit.oak.plugins.segment.Record.getSegment(Record.java:82)
at org.apache.jackrabbit.oak.plugins.segment.MapRecord.size(MapRecord.java:142)
at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeMapBucket(SegmentWriter.java:534)
- locked <0x00000005c0c57640> (a org.apache.jackrabbit.oak.plugins.segment.SegmentWriter)
at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeMapBucket(SegmentWriter.java:532)
- locked <0x00000005c0c57640> (a org.apache.jackrabbit.oak.plugins.segment.SegmentWriter)
at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeMapBucket(SegmentWriter.java:532)
- locked <0x00000005c0c57640> (a org.apache.jackrabbit.oak.plugins.segment.SegmentWriter)
at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeMap(SegmentWriter.java:784)
at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1172)
at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1174)
at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1155)
at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:399)
at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1146)
at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1155)
at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:399)
at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1146)
at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.getNodeState(SegmentNodeBuilder.java:100)
at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.updated(SegmentNodeBuilder.java:85)
at org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.updated(MemoryNodeBuilder.java:214)
at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.updated(SegmentNodeBuilder.java:81)
at org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.remove(MemoryNodeBuilder.java:355)
at org.apache.jackrabbit.oak.plugins.index.property.strategy.OrderedContentMirrorStoreStrategy.prune(OrderedContentMirrorStoreStrategy.java:242)
at org.apache.jackrabbit.oak.plugins.index.property.strategy.ContentMirrorStoreStrategy.remove(ContentMirrorStoreStrategy.java:118)
at org.apache.jackrabbit.oak.plugins.index.property.strategy.ContentMirrorStoreStrategy.update(ContentMirrorStoreStrategy.java:90)
at org.apache.jackrabbit.oak.plugins.index.property.PropertyIndexEditor.leave(PropertyIndexEditor.java:274)
at org.apache.jackrabbit.oak.plugins.index.property.OrderedPropertyIndexEditor.leave(OrderedPropertyIndexEditor.java:176)
at org.apache.jackrabbit.oak.spi.commit.CompositeEditor.leave(CompositeEditor.java:74)
at org.apache.jackrabbit.oak.spi.commit.VisibleEditor.leave(VisibleEditor.java:63)
at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeDeleted(EditorDiff.java:176)
at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstBaseState(EmptyNodeState.java:143)
at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeDeleted(EditorDiff.java:171)
at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstBaseState(EmptyNodeState.java:143)
at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeDeleted(EditorDiff.java:171)
at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstBaseState(EmptyNodeState.java:143)
at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeDeleted(EditorDiff.java:171)
at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstBaseState(EmptyNodeState.java:143)
at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeDeleted(EditorDiff.java:171)
at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstBaseState(EmptyNodeState.java:143)
at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeDeleted(EditorDiff.java:171)
at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstBaseState(EmptyNodeState.java:143)
at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeDeleted(EditorDiff.java:171)
at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstBaseState(EmptyNodeState.java:143)
at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeDeleted(EditorDiff.java:171)
at org.apache.jackrabbit.oak.plugins.segment.MapRecord$3.childNodeDeleted(MapRecord.java:449)
at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:481)
at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:436)
at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:583)
at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148)
at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:531)
at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148)
at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:418)
at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:583)
at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148)
at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:418)
at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:583)
at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148)
at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:418)
at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:583)
at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148)
at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:418)
at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:583)
at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148)
at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:418)
at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:583)
at org.apache.jackrabbit.oak.spi.commit.EditorDiff.process(EditorDiff.java:52)
at org.apache.jackrabbit.oak.spi.commit.EditorHook.processCommit(EditorHook.java:54)
at org.apache.jackrabbit.oak.spi.commit.CompositeHook.processCommit(CompositeHook.java:61)
at org.apache.jackrabbit.oak.spi.commit.CompositeHook.processCommit(CompositeHook.java:61)
at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore$Commit.prepare(SegmentNodeStore.java:458)
at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore$Commit.optimisticMerge(SegmentNodeStore.java:489)
at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore$Commit.execute(SegmentNodeStore.java:545)
at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore.merge(SegmentNodeStore.java:208)
at org.apache.jackrabbit.oak.spi.state.ProxyNodeStore.merge(ProxyNodeStore.java:43)
at org.apache.jackrabbit.oak.core.MutableRoot.commit(MutableRoot.java:247)
at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.commit(SessionDelegate.java:339)
at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.commit(SessionDelegate.java:364)
at org.apache.jackrabbit.oak.jcr.version.ReadWriteVersionManager.removeVersion(ReadWriteVersionManager.java:239)
at org.apache.jackrabbit.oak.jcr.delegate.VersionManagerDelegate.removeVersion(VersionManagerDelegate.java:226)
at org.apache.jackrabbit.oak.jcr.delegate.VersionHistoryDelegate.removeVersion(VersionHistoryDelegate.java:209)
at org.apache.jackrabbit.oak.jcr.version.VersionHistoryImpl$11.performVoid(VersionHistoryImpl.java:240)
at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.performVoid(SessionDelegate.java:266)
at org.apache.jackrabbit.oak.jcr.version.VersionHistoryImpl.removeVersion(VersionHistoryImpl.java:236)
at com.day.cq.wcm.core.impl.VersionManagerImpl.purgeVersions(VersionManagerImpl.java:380)
at com.day.cq.wcm.core.impl.VersionManagerImpl.purgeVersions(VersionManagerImpl.java:346)
at com.day.cq.wcm.core.impl.VersionManagerImpl.purgeVersions(VersionManagerImpl.java:338)
at com.day.cq.wcm.core.impl.VersionManagerImpl.purgeVersions(VersionManagerImpl.java:569)
at com.day.cq.wcm.core.impl.VersionManagerImpl.onEvent(VersionManagerImpl.java:329)
at org.apache.jackrabbit.commons.observation.ListenerTracker$1.onEvent(ListenerTracker.java:164)
at org.apache.jackrabbit.oak.jcr.observation.ChangeProcessor.contentChanged(ChangeProcessor.java:316)
at org.apache.jackrabbit.oak.spi.commit.BackgroundObserver$1$1.call(BackgroundObserver.java:131)
at org.apache.jackrabbit.oak.spi.commit.BackgroundObserver$1$1.call(BackgroundObserver.java:125)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000005c23fd400> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x00000005c24ad9d0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x00000005c6039780> (a java.util.concurrent.ThreadPoolExecutor$Worker)