Description

Hi,

I’m a bit stuck here and needs a clue or two in order to continue our investigations. Hope that someone can help.

Periodically, around once a month, we get the below ArrayIndexOutOfBoundsException on our system. We use multiple indexes and the error can originate from any of them, but the error always occurs in line 1217 in BKDWriter (during a System.arraycopy).

We found a couple of issues on the net regarding JIT optimization problem related to J9, but they all looks like they have been resolved and cannot be reproduced anymore. But nevertheless, we have added the following -Xjit flags to excludes JIT optimization for every class / inner classes in the bkd package. Moreover, we have also made a complete copy of the whole installation in production, and added the opposite arguments (enforce JIT optimizations for the specific classes). First we will try with optLevel=hot, but if this doesn’t show anything we will afterwards try with veryHot and scorching. Unfortunately we do not have the result of this yet, but I’ll of course post it when it is known.

Unfortunately it’s not possible to run OpenJDK on the IBM i platform, so such a test will not be possible, but it is worth mentioning that our product is a standard product, which typically run on the Windows or Linux platform using AdoptOpenJDK. Currently we have a couple of hundred installations running out there on these platforms, and without any problems. But on the IBM I platform with J9 we sometimes see this exception.

Any good ideas for further investigations? Or have seen such issue before?

We are using Lucene 6.6.0 and runs on IBM J9 on the IBM I platform.

Stacktrace

Exception in thread "Lucene Merge Thread #0" 2019-05-01T06:10:07.970 CEST [Lucene Merge Thread #0] org.apache.lucene.index.MergePolicy$MergeException: java.lang.ArrayIndexOutOfBoundsException
at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:703)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:683)
Caused by: 2019-05-01T06:10:07.971 CEST [Lucene Merge Thread #0] java.lang.ArrayIndexOutOfBoundsException
at org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1217)
at org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1197)
at org.apache.lucene.util.bkd.BKDWriter.packIndex(BKDWriter.java:1078)
at org.apache.lucene.util.bkd.BKDWriter.writeIndex(BKDWriter.java:1245)
at org.apache.lucene.util.bkd.BKDWriter.access$600(BKDWriter.java:82)
at org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.finish(BKDWriter.java:648)
at org.apache.lucene.util.bkd.BKDWriter.merge(BKDWriter.java:560)
at org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.merge(Lucene60PointsWriter.java:212)
at org.apache.lucene.index.SegmentMerger.mergePoints(SegmentMerger.java:173)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:122)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4356)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)
Exception in thread "Lucene Merge Thread #0" 2019-05-01T06:10:08.075 CEST [Lucene Merge Thread #0] org.apache.lucene.index.MergePolicy$MergeException: java.lang.ArrayIndexOutOfBoundsException
at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:703)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:683)
Caused by: 2019-05-01T06:10:08.076 CEST [Lucene Merge Thread #0] java.lang.ArrayIndexOutOfBoundsException
at org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1217)
at org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1197)
at org.apache.lucene.util.bkd.BKDWriter.packIndex(BKDWriter.java:1078)
at org.apache.lucene.util.bkd.BKDWriter.writeIndex(BKDWriter.java:1245)
at org.apache.lucene.util.bkd.BKDWriter.access$600(BKDWriter.java:82)
at org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.finish(BKDWriter.java:648)
at org.apache.lucene.util.bkd.BKDWriter.merge(BKDWriter.java:560)
at org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.merge(Lucene60PointsWriter.java:212)
at org.apache.lucene.index.SegmentMerger.mergePoints(SegmentMerger.java:173)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:122)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4356)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler

On production we have disabled jit optimization for the following classes