Description

Hi

Following exception occurs while running our application with embedded derby database. The application uses multithreading. This exception occurs while insert query is executed. The insert query is run using JDBC:

There are multiple threads inserting data in database. I tried to search a lot on net but could not find any solution. We're using derby 10.3.1.4. Not sure about the root cause. It will be great if anybody provides some solution because it is creating lot of problems in our application.

Please try your testing using the 10.3 fixpack. A very serious bug involving container (e.g. file) handling was fixed: DERBY-3347. The bug produced a number of errors - many of which caused corruption. Please run a consistency check on your database to insure no corrupt information was written to disk. The NPE you experience did occur in container handling code though I am not familiar enough with the code to know if this sort of error could have been caused by DERBY-3347.

First exeception on the stack you reported : openContainer()
Caused by: java.lang.NullPointerException
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(Unknown Source)
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(Unknown Source)
at org.apache.derby.impl.store.raw.xact.Xact.openContainer(Unknown

A bug that could cause unrecoverable database corruption has been fixed.
Symptoms Seen by Applications Affected by Change

A bug that could cause database corruption was introduced in the 10.3 codeline and affects the following releases:

Apache Derby 10.3.1.4

Apache Derby 10.3.2.1

Users who are hit by this bug may experience exceptions at various times during execution of SQL statements, booting or shutdown of a database, or during checkpointing. It may result in a number of different error messages, including any of these:

ERROR XSDB3: Container information cannot change once written: was 0, now 80
ERROR XSDG1: Page Page(1039,Container(0, 5856)) could not be written to disk, please check if disk is full.
ERROR XSDG2: Invalid checksum on Page Page(0,Container(0, 1313))
ERROR XSDG3: Meta-data for Container org.apache.derby.impl.store.raw.data.RAFContainer4@1afb0c7 could not be accessed
ERROR XSLA1: Log Record has been sent to the stream, but it cannot be applied to the store (Object null). This may cause recovery problems also.

Stan Bradbury
added a comment - 22/Jul/08 00:41 Please try your testing using the 10.3 fixpack. A very serious bug involving container (e.g. file) handling was fixed: DERBY-3347 . The bug produced a number of errors - many of which caused corruption. Please run a consistency check on your database to insure no corrupt information was written to disk. The NPE you experience did occur in container handling code though I am not familiar enough with the code to know if this sort of error could have been caused by DERBY-3347 .
First exeception on the stack you reported : openContainer()
Caused by: java.lang.NullPointerException
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(Unknown Source)
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(Unknown Source)
at org.apache.derby.impl.store.raw.xact.Xact.openContainer(Unknown
From Release Notes - DERBY-3347 :
A bug that could cause unrecoverable database corruption has been fixed.
Symptoms Seen by Applications Affected by Change
A bug that could cause database corruption was introduced in the 10.3 codeline and affects the following releases:
Apache Derby 10.3.1.4
Apache Derby 10.3.2.1
Users who are hit by this bug may experience exceptions at various times during execution of SQL statements, booting or shutdown of a database, or during checkpointing. It may result in a number of different error messages, including any of these:
ERROR XSDB3: Container information cannot change once written: was 0, now 80
ERROR XSDG1: Page Page(1039,Container(0, 5856)) could not be written to disk, please check if disk is full.
ERROR XSDG2: Invalid checksum on Page Page(0,Container(0, 1313))
ERROR XSDG3: Meta-data for Container org.apache.derby.impl.store.raw.data.RAFContainer4@1afb0c7 could not be accessed
ERROR XSLA1: Log Record has been sent to the stream, but it cannot be applied to the store (Object null). This may cause recovery problems also.
HTH

Thanks for providing me useful inputs regarding the problem. In our
application there are mutilple threads performing db operations
constantly, so updating the jars may address the strange issues we
face. Also perfomance is very crucial for our application. I'll try
using the latest fixpack either 10.3.3.0 or 10.4.1.0 and get back with
the results regarding the errors and performance.

vibhuti gupta
added a comment - 23/Jul/08 06:47 Thanks for providing me useful inputs regarding the problem. In our
application there are mutilple threads performing db operations
constantly, so updating the jars may address the strange issues we
face. Also perfomance is very crucial for our application. I'll try
using the latest fixpack either 10.3.3.0 or 10.4.1.0 and get back with
the results regarding the errors and performance.

We have seen this issue at a customer deployment using 10.4.1.3 under Windows - not sure which OS version. Here is the stack trace with line numbers. The root exception is the same as what has been reported in this bug.

Caused by: java.sql.SQLException: Java exception: ': java.lang.NullPointerException'.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:45)
at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:87)
at org.apache.derby.impl.jdbc.Util.javaException(Util.java:244)
at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(TransactionResourceImpl.java:403)
at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(TransactionResourceImpl.java:346)
at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:2125)
at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:571)
at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(EmbedConnection30.java:73)
at org.apache.derby.jdbc.Driver30.getNewEmbedConnection(Driver30.java:80)
at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:238)
at org.apache.derby.jdbc.EmbeddedDataSource.getConnection(EmbeddedDataSource.java:479)
at org.apache.derby.jdbc.EmbedPooledConnection.openRealConnection(EmbedPooledConnection.java:171)
at org.apache.derby.jdbc.EmbedPooledConnection.<init>(EmbedPooledConnection.java:113)
at org.apache.derby.jdbc.Driver30.getNewPooledConnection(Driver30.java:141)
at org.apache.derby.jdbc.EmbeddedConnectionPoolDataSource.createPooledConnection(EmbeddedConnectionPoolDataSource.java:129)
at org.apache.derby.jdbc.EmbeddedConnectionPoolDataSource.getPooledConnection(EmbeddedConnectionPoolDataSource.java:75)
at bK.makeObject(SourceFile:163)
at org.apache.commons.pool.impl.StackObjectPool.borrowObject(StackObjectPool.java:131)
at gh.a(SourceFile:86)
at vH.a(SourceFile:101)
at vH.a(SourceFile:157)
at FO.b(SourceFile:74)
... 40 more

Caused by: java.lang.NullPointerException
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(BaseDataFileFactory.java:629)
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(BaseDataFileFactory.java:559)
at org.apache.derby.impl.store.raw.xact.Xact.openContainer(Xact.java:1283)
at org.apache.derby.impl.store.access.conglomerate.OpenConglomerate.init(OpenConglomerate.java:863)
at org.apache.derby.impl.store.access.heap.Heap.open(Heap.java:683)
at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:479)
at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:1311)
at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndex(DataDictionaryImpl.java:7907)
at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.locateSchemaRow(DataDictionaryImpl.java:1556)
at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getSchemaDescriptor(DataDictionaryImpl.java:1466)
at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.initDefaultSchemaDescriptor(GenericLanguageConnectionContext.java:363)
at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.initialize(GenericLanguageConnectionContext.java:345)
at org.apache.derby.impl.db.BasicDatabase.setupConnection(BasicDatabase.java:309)
at org.apache.derby.impl.jdbc.TransactionResourceImpl.startTransaction(TransactionResourceImpl.java:188)
at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:416)
... 55 more

David Sitsky
added a comment - 11/Aug/08 00:43 We have seen this issue at a customer deployment using 10.4.1.3 under Windows - not sure which OS version. Here is the stack trace with line numbers. The root exception is the same as what has been reported in this bug.
Caused by: java.sql.SQLException: Java exception: ': java.lang.NullPointerException'.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:45)
at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:87)
at org.apache.derby.impl.jdbc.Util.javaException(Util.java:244)
at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(TransactionResourceImpl.java:403)
at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(TransactionResourceImpl.java:346)
at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:2125)
at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:571)
at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(EmbedConnection30.java:73)
at org.apache.derby.jdbc.Driver30.getNewEmbedConnection(Driver30.java:80)
at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:238)
at org.apache.derby.jdbc.EmbeddedDataSource.getConnection(EmbeddedDataSource.java:479)
at org.apache.derby.jdbc.EmbedPooledConnection.openRealConnection(EmbedPooledConnection.java:171)
at org.apache.derby.jdbc.EmbedPooledConnection.<init>(EmbedPooledConnection.java:113)
at org.apache.derby.jdbc.Driver30.getNewPooledConnection(Driver30.java:141)
at org.apache.derby.jdbc.EmbeddedConnectionPoolDataSource.createPooledConnection(EmbeddedConnectionPoolDataSource.java:129)
at org.apache.derby.jdbc.EmbeddedConnectionPoolDataSource.getPooledConnection(EmbeddedConnectionPoolDataSource.java:75)
at bK.makeObject(SourceFile:163)
at org.apache.commons.pool.impl.StackObjectPool.borrowObject(StackObjectPool.java:131)
at gh.a(SourceFile:86)
at vH.a(SourceFile:101)
at vH.a(SourceFile:157)
at FO.b(SourceFile:74)
... 40 more
Caused by: java.lang.NullPointerException
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(BaseDataFileFactory.java:629)
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(BaseDataFileFactory.java:559)
at org.apache.derby.impl.store.raw.xact.Xact.openContainer(Xact.java:1283)
at org.apache.derby.impl.store.access.conglomerate.OpenConglomerate.init(OpenConglomerate.java:863)
at org.apache.derby.impl.store.access.heap.Heap.open(Heap.java:683)
at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:479)
at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(RAMTransaction.java:1311)
at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndex(DataDictionaryImpl.java:7907)
at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.locateSchemaRow(DataDictionaryImpl.java:1556)
at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getSchemaDescriptor(DataDictionaryImpl.java:1466)
at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.initDefaultSchemaDescriptor(GenericLanguageConnectionContext.java:363)
at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.initialize(GenericLanguageConnectionContext.java:345)
at org.apache.derby.impl.db.BasicDatabase.setupConnection(BasicDatabase.java:309)
at org.apache.derby.impl.jdbc.TransactionResourceImpl.startTransaction(TransactionResourceImpl.java:188)
at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:416)
... 55 more

Whatever is causing this issue, it appears to be transient in nature. We advised our customer to restart the application, which of course closes and re-opens the Derby database, and apparently the above problem didn't surface this time. So it is not entirely clear what is triggering this condition.

David Sitsky
added a comment - 12/Aug/08 00:22 Whatever is causing this issue, it appears to be transient in nature. We advised our customer to restart the application, which of course closes and re-opens the Derby database, and apparently the above problem didn't surface this time. So it is not entirely clear what is triggering this condition.

In our case this is being caused by a database stored on a file server.

At some point in time, the network goes out for a moment, resulting in Derby recording a corruption error. Subsequent queries then receive a NullPointerException as some field is set to null in this case, and then closing and reopening the database makes it work again because the database itself has not actually been corrupted.

So the problem appears to be twofold:
1. We're getting a NPE instead of a exception saying something about corruption or an inability to read from the store.
2. Derby is assuming that a failure reading from the store now guarantees a failure to read from the store later, which is not the case for a file store where the files are on another computer being accessed over the network.

Trejkaz
added a comment - 02/Oct/08 03:52 In our case this is being caused by a database stored on a file server.
At some point in time, the network goes out for a moment, resulting in Derby recording a corruption error. Subsequent queries then receive a NullPointerException as some field is set to null in this case, and then closing and reopening the database makes it work again because the database itself has not actually been corrupted.
So the problem appears to be twofold:
1. We're getting a NPE instead of a exception saying something about corruption or an inability to read from the store.
2. Derby is assuming that a failure reading from the store now guarantees a failure to read from the store later, which is not the case for a file store where the files are on another computer being accessed over the network.

Kathey Marsden
added a comment - 30/Oct/08 21:32 The documentation says:
You can specify only databases that are local to the machine on which the JVM is running. NFS file systems on UNIX and remote shared files on Windows (//machine/directory) are not guaranteed to work.
see:
http://db.apache.org/derby/docs/dev/devguide/cdevdvlp40350.html
I think the corruption you are seeing is because of the network access to the files. If that is the case I think this issue can be closed as invalid.

Surely when the network goes back to normal, the database should work again. It seems silly to have to shut down the database and reopen it just to get Derby back into the correct state again, not to mention that it's more or less impossible to do this under Network Server.

Trejkaz
added a comment - 30/Oct/08 22:42 Surely when the network goes back to normal, the database should work again. It seems silly to have to shut down the database and reopen it just to get Derby back into the correct state again, not to mention that it's more or less impossible to do this under Network Server.

Kathey Marsden
added a comment - 30/Oct/08 22:59 I'm afraid not. Derby cannot guarantee the consistency of the database when accessed over the network. It is a common cause for corruption and should not be done.

You appear to be dodging the issue. In cases where we have seen this, the database is not corrupt, Derby just thinks it is until it is forced to reopen it. When it reopens it, it works fine, so why can't it do this under the covers?

Trejkaz
added a comment - 30/Oct/08 23:05 You appear to be dodging the issue. In cases where we have seen this, the database is not corrupt, Derby just thinks it is until it is forced to reopen it. When it reopens it, it works fine, so why can't it do this under the covers?

Well I guess if I had my preference we would have a way to prevent network access of databases completely or find a way to support it. Unfortunately a feasible solution on either count has not presented itself. I wouldn't want to do anything "under the covers" to promote this bad practice. I am glad to hear your database is not yet corrupt. I hope that is indeed the case.

Kathey Marsden
added a comment - 31/Oct/08 00:54 Well I guess if I had my preference we would have a way to prevent network access of databases completely or find a way to support it. Unfortunately a feasible solution on either count has not presented itself. I wouldn't want to do anything "under the covers" to promote this bad practice. I am glad to hear your database is not yet corrupt. I hope that is indeed the case.

I don't think this really comes down to network vs. non-network. It should come down to a "usable filesystem", and I haven't yet heard a good definition of what constitutes a "usable filesystem". On the mailing list, it was suggested that reliable sync() and locking were the only requirements. As far as I know CIFS and NFS can do both of these, which would suggest that it's just as safe to use a network as a locally attached disk.

Networks can go down temporarily, sure, but so can "directly" attached disks (e.g. eSATA, or hot-swappable drives. What about USB and FireWire?)

So it seems to me this particular problem could very well occur if a directly attached disk disappeared and then later reappeared. It's not really an issue of corruption at all... or so you'd hope, otherwise Derby wouldn't be resilient to a power outage either.

Trejkaz
added a comment - 01/Nov/08 13:10 - edited I don't think this really comes down to network vs. non-network. It should come down to a "usable filesystem", and I haven't yet heard a good definition of what constitutes a "usable filesystem". On the mailing list, it was suggested that reliable sync() and locking were the only requirements. As far as I know CIFS and NFS can do both of these, which would suggest that it's just as safe to use a network as a locally attached disk.
Networks can go down temporarily, sure, but so can "directly" attached disks (e.g. eSATA, or hot-swappable drives. What about USB and FireWire?)
So it seems to me this particular problem could very well occur if a directly attached disk disappeared and then later reappeared. It's not really an issue of corruption at all... or so you'd hope, otherwise Derby wouldn't be resilient to a power outage either.

David Sitsky
added a comment - 11/Nov/08 03:42 I've seen this issue happen again with 10.4.1.3 under Windows, with the same stack-trace as reported earlier ( https://issues.apache.org/jira/browse/DERBY-3746?focusedCommentId=12621320#action_12621320 ). Despite the discussions on this bug about network filesystems, my incidents happen on a local filesystem with an embedded derby connection - no derby server is involved at all.
Does anyone have any ideas about this? I know shutting down the program and restarting fixes the issue, but it doesn't look very good for the end-user..
All I know is there must be some sort of race condition occurring, as it is hard to reproduce this.

To provide more information - I've attached some more stack traces from the derby.log file itself. In our application, we have multiple threads, each with their own embedded derby connection object potentially accessing the same blob of data concurrently. Its unfortunately very difficult to reproduce this issue.

------------ BEGIN SHUTDOWN ERROR STACK -------------

ERROR XSDG0: Page Page(43,Container(0, 1344)) could not be read from disk.
at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:336)
at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.java:690)
at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(CachedPage.java:190)
at org.apache.derby.impl.services.cache.ConcurrentCache.find(ConcurrentCache.java:295)
at org.apache.derby.impl.store.raw.data.FileContainer.getUserPage(FileContainer.java:2432)
at org.apache.derby.impl.store.raw.data.FileContainer.getPage(FileContainer.java:2482)
at org.apache.derby.impl.store.raw.data.BaseContainerHandle.getPage(BaseContainerHandle.java:319)
at org.apache.derby.impl.store.access.conglomerate.OpenConglomerate.latchPage(OpenConglomerate.java:294)
at org.apache.derby.impl.store.access.conglomerate.GenericConglomerateController.fetch(GenericConglomerateController.java:263)
at org.apache.derby.impl.sql.execute.IndexRowToBaseRowResultSet.getNextRowCore(IndexRowToBaseRowResultSet.java:389)
at org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(ProjectRestrictResultSet.java:255)
at org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getNextRow(BasicNoPutResultSetImpl.java:460)
at org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(EmbedResultSet.java:423)
at org.apache.derby.impl.jdbc.EmbedResultSet.next(EmbedResultSet.java:367)
at Pu.a(SourceFile:305)
at Pu.a(SourceFile:292)
at IT.a(SourceFile:118)
at ya.a(SourceFile:160)
at IT.b(SourceFile:73)
at qy.read(SourceFile:291)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at java.util.zip.CheckedInputStream.read(CheckedInputStream.java:42)
at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:205)
at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:197)
at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:136)
at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:58)
at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:68)
at uP.a(SourceFile:94)
at LT.a(SourceFile:78)
at Kq.a(SourceFile:55)
at cE.a(SourceFile:229)
at com.nuix.util.e.a(SourceFile:93)
at com.nuix.util.e.read(SourceFile:40)
at org.apache.lucene.analysis.standard.StandardTokenizerImpl.zzRefill(StandardTokenizerImpl.java:337)
at org.apache.lucene.analysis.standard.StandardTokenizerImpl.getNextToken(StandardTokenizerImpl.java:523)
at org.apache.lucene.analysis.standard.StandardTokenizer.next(StandardTokenizer.java:139)
at org.apache.lucene.analysis.standard.StandardFilter.next(StandardFilter.java:41)
at org.apache.lucene.analysis.TokenStream.next(TokenStream.java:45)
at NF.next(SourceFile:48)
at org.apache.lucene.analysis.TokenStream.next(TokenStream.java:79)
at org.apache.lucene.analysis.LowerCaseFilter.next(LowerCaseFilter.java:33)
at org.apache.lucene.analysis.TokenStream.next(TokenStream.java:45)
at org.apache.lucene.search.similar.MoreLikeThis.addTermFrequencies(MoreLikeThis.java:836)
at org.apache.lucene.search.similar.MoreLikeThis.retrieveTerms(MoreLikeThis.java:906)
at org.apache.lucene.search.similar.MoreLikeThis.retrieveInterestingTerms(MoreLikeThis.java:922)
at gm.a(SourceFile:245)
at gm.a(SourceFile:286)
at Hq.<init>(SourceFile:243)
at Hq.<init>(SourceFile:230)
at rq.a(SourceFile:175)
at rq.doInBackground(SourceFile:147)
at javax.swing.SwingWorker$1.call(SwingWorker.java:278)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at javax.swing.SwingWorker.run(SwingWorker.java:317)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.nio.channels.ClosedChannelException
at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:91)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:616)
at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(RAFContainer4.java:471)
at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage0(RAFContainer4.java:242)
at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(RAFContainer4.java:212)
at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.java:666)
... 57 more
============= begin nested exception, level (1) ===========
java.nio.channels.ClosedChannelException
at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:91)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:616)
at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(RAFContainer4.java:471)
at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage0(RAFContainer4.java:242)
at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(RAFContainer4.java:212)
at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.java:666)
at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(CachedPage.java:190)
at org.apache.derby.impl.services.cache.ConcurrentCache.find(ConcurrentCache.java:295)
at org.apache.derby.impl.store.raw.data.FileContainer.getUserPage(FileContainer.java:2432)
at org.apache.derby.impl.store.raw.data.FileContainer.getPage(FileContainer.java:2482)
at org.apache.derby.impl.store.raw.data.BaseContainerHandle.getPage(BaseContainerHandle.java:319)
at org.apache.derby.impl.store.access.conglomerate.OpenConglomerate.latchPage(OpenConglomerate.java:294)
at org.apache.derby.impl.store.access.conglomerate.GenericConglomerateController.fetch(GenericConglomerateController.java:263)
at org.apache.derby.impl.sql.execute.IndexRowToBaseRowResultSet.getNextRowCore(IndexRowToBaseRowResultSet.java:389)
at org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(ProjectRestrictResultSet.java:255)
at org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getNextRow(BasicNoPutResultSetImpl.java:460)
at org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(EmbedResultSet.java:423)
at org.apache.derby.impl.jdbc.EmbedResultSet.next(EmbedResultSet.java:367)
at Pu.a(SourceFile:305)
at Pu.a(SourceFile:292)
at IT.a(SourceFile:118)
at ya.a(SourceFile:160)
at IT.b(SourceFile:73)
at qy.read(SourceFile:291)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at java.util.zip.CheckedInputStream.read(CheckedInputStream.java:42)
at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:205)
at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:197)
at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:136)
at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:58)
at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:68)
at uP.a(SourceFile:94)
at LT.a(SourceFile:78)
at Kq.a(SourceFile:55)
at cE.a(SourceFile:229)
at com.nuix.util.e.a(SourceFile:93)
at com.nuix.util.e.read(SourceFile:40)
at org.apache.lucene.analysis.standard.StandardTokenizerImpl.zzRefill(StandardTokenizerImpl.java:337)
at org.apache.lucene.analysis.standard.StandardTokenizerImpl.getNextToken(StandardTokenizerImpl.java:523)
at org.apache.lucene.analysis.standard.StandardTokenizer.next(StandardTokenizer.java:139)
at org.apache.lucene.analysis.standard.StandardFilter.next(StandardFilter.java:41)
at org.apache.lucene.analysis.TokenStream.next(TokenStream.java:45)
at NF.next(SourceFile:48)
at org.apache.lucene.analysis.TokenStream.next(TokenStream.java:79)
at org.apache.lucene.analysis.LowerCaseFilter.next(LowerCaseFilter.java:33)
at org.apache.lucene.analysis.TokenStream.next(TokenStream.java:45)
at org.apache.lucene.search.similar.MoreLikeThis.addTermFrequencies(MoreLikeThis.java:836)
at org.apache.lucene.search.similar.MoreLikeThis.retrieveTerms(MoreLikeThis.java:906)
at org.apache.lucene.search.similar.MoreLikeThis.retrieveInterestingTerms(MoreLikeThis.java:922)
at gm.a(SourceFile:245)
at gm.a(SourceFile:286)
at Hq.<init>(SourceFile:243)
at Hq.<init>(SourceFile:230)
at rq.a(SourceFile:175)
at rq.doInBackground(SourceFile:147)
at javax.swing.SwingWorker$1.call(SwingWorker.java:278)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at javax.swing.SwingWorker.run(SwingWorker.java:317)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
============= end nested exception, level (1) ===========

David Sitsky
added a comment - 11/Nov/08 05:09 To provide more information - I've attached some more stack traces from the derby.log file itself. In our application, we have multiple threads, each with their own embedded derby connection object potentially accessing the same blob of data concurrently. Its unfortunately very difficult to reproduce this issue.
------------ BEGIN SHUTDOWN ERROR STACK -------------
ERROR XSDG0: Page Page(43,Container(0, 1344)) could not be read from disk.
at org.apache.derby.iapi.error.StandardException.newException(StandardException.java:336)
at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.java:690)
at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(CachedPage.java:190)
at org.apache.derby.impl.services.cache.ConcurrentCache.find(ConcurrentCache.java:295)
at org.apache.derby.impl.store.raw.data.FileContainer.getUserPage(FileContainer.java:2432)
at org.apache.derby.impl.store.raw.data.FileContainer.getPage(FileContainer.java:2482)
at org.apache.derby.impl.store.raw.data.BaseContainerHandle.getPage(BaseContainerHandle.java:319)
at org.apache.derby.impl.store.access.conglomerate.OpenConglomerate.latchPage(OpenConglomerate.java:294)
at org.apache.derby.impl.store.access.conglomerate.GenericConglomerateController.fetch(GenericConglomerateController.java:263)
at org.apache.derby.impl.sql.execute.IndexRowToBaseRowResultSet.getNextRowCore(IndexRowToBaseRowResultSet.java:389)
at org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(ProjectRestrictResultSet.java:255)
at org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getNextRow(BasicNoPutResultSetImpl.java:460)
at org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(EmbedResultSet.java:423)
at org.apache.derby.impl.jdbc.EmbedResultSet.next(EmbedResultSet.java:367)
at Pu.a(SourceFile:305)
at Pu.a(SourceFile:292)
at IT.a(SourceFile:118)
at ya.a(SourceFile:160)
at IT.b(SourceFile:73)
at qy.read(SourceFile:291)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at java.util.zip.CheckedInputStream.read(CheckedInputStream.java:42)
at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:205)
at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:197)
at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:136)
at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:58)
at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:68)
at uP.a(SourceFile:94)
at LT.a(SourceFile:78)
at Kq.a(SourceFile:55)
at cE.a(SourceFile:229)
at com.nuix.util.e.a(SourceFile:93)
at com.nuix.util.e.read(SourceFile:40)
at org.apache.lucene.analysis.standard.StandardTokenizerImpl.zzRefill(StandardTokenizerImpl.java:337)
at org.apache.lucene.analysis.standard.StandardTokenizerImpl.getNextToken(StandardTokenizerImpl.java:523)
at org.apache.lucene.analysis.standard.StandardTokenizer.next(StandardTokenizer.java:139)
at org.apache.lucene.analysis.standard.StandardFilter.next(StandardFilter.java:41)
at org.apache.lucene.analysis.TokenStream.next(TokenStream.java:45)
at NF.next(SourceFile:48)
at org.apache.lucene.analysis.TokenStream.next(TokenStream.java:79)
at org.apache.lucene.analysis.LowerCaseFilter.next(LowerCaseFilter.java:33)
at org.apache.lucene.analysis.TokenStream.next(TokenStream.java:45)
at org.apache.lucene.search.similar.MoreLikeThis.addTermFrequencies(MoreLikeThis.java:836)
at org.apache.lucene.search.similar.MoreLikeThis.retrieveTerms(MoreLikeThis.java:906)
at org.apache.lucene.search.similar.MoreLikeThis.retrieveInterestingTerms(MoreLikeThis.java:922)
at gm.a(SourceFile:245)
at gm.a(SourceFile:286)
at Hq.<init>(SourceFile:243)
at Hq.<init>(SourceFile:230)
at rq.a(SourceFile:175)
at rq.doInBackground(SourceFile:147)
at javax.swing.SwingWorker$1.call(SwingWorker.java:278)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at javax.swing.SwingWorker.run(SwingWorker.java:317)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.nio.channels.ClosedChannelException
at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:91)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:616)
at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(RAFContainer4.java:471)
at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage0(RAFContainer4.java:242)
at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(RAFContainer4.java:212)
at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.java:666)
... 57 more
============= begin nested exception, level (1) ===========
java.nio.channels.ClosedChannelException
at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:91)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:616)
at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(RAFContainer4.java:471)
at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage0(RAFContainer4.java:242)
at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(RAFContainer4.java:212)
at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.java:666)
at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(CachedPage.java:190)
at org.apache.derby.impl.services.cache.ConcurrentCache.find(ConcurrentCache.java:295)
at org.apache.derby.impl.store.raw.data.FileContainer.getUserPage(FileContainer.java:2432)
at org.apache.derby.impl.store.raw.data.FileContainer.getPage(FileContainer.java:2482)
at org.apache.derby.impl.store.raw.data.BaseContainerHandle.getPage(BaseContainerHandle.java:319)
at org.apache.derby.impl.store.access.conglomerate.OpenConglomerate.latchPage(OpenConglomerate.java:294)
at org.apache.derby.impl.store.access.conglomerate.GenericConglomerateController.fetch(GenericConglomerateController.java:263)
at org.apache.derby.impl.sql.execute.IndexRowToBaseRowResultSet.getNextRowCore(IndexRowToBaseRowResultSet.java:389)
at org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(ProjectRestrictResultSet.java:255)
at org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getNextRow(BasicNoPutResultSetImpl.java:460)
at org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(EmbedResultSet.java:423)
at org.apache.derby.impl.jdbc.EmbedResultSet.next(EmbedResultSet.java:367)
at Pu.a(SourceFile:305)
at Pu.a(SourceFile:292)
at IT.a(SourceFile:118)
at ya.a(SourceFile:160)
at IT.b(SourceFile:73)
at qy.read(SourceFile:291)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at java.util.zip.CheckedInputStream.read(CheckedInputStream.java:42)
at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:205)
at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:197)
at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:136)
at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:58)
at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:68)
at uP.a(SourceFile:94)
at LT.a(SourceFile:78)
at Kq.a(SourceFile:55)
at cE.a(SourceFile:229)
at com.nuix.util.e.a(SourceFile:93)
at com.nuix.util.e.read(SourceFile:40)
at org.apache.lucene.analysis.standard.StandardTokenizerImpl.zzRefill(StandardTokenizerImpl.java:337)
at org.apache.lucene.analysis.standard.StandardTokenizerImpl.getNextToken(StandardTokenizerImpl.java:523)
at org.apache.lucene.analysis.standard.StandardTokenizer.next(StandardTokenizer.java:139)
at org.apache.lucene.analysis.standard.StandardFilter.next(StandardFilter.java:41)
at org.apache.lucene.analysis.TokenStream.next(TokenStream.java:45)
at NF.next(SourceFile:48)
at org.apache.lucene.analysis.TokenStream.next(TokenStream.java:79)
at org.apache.lucene.analysis.LowerCaseFilter.next(LowerCaseFilter.java:33)
at org.apache.lucene.analysis.TokenStream.next(TokenStream.java:45)
at org.apache.lucene.search.similar.MoreLikeThis.addTermFrequencies(MoreLikeThis.java:836)
at org.apache.lucene.search.similar.MoreLikeThis.retrieveTerms(MoreLikeThis.java:906)
at org.apache.lucene.search.similar.MoreLikeThis.retrieveInterestingTerms(MoreLikeThis.java:922)
at gm.a(SourceFile:245)
at gm.a(SourceFile:286)
at Hq.<init>(SourceFile:243)
at Hq.<init>(SourceFile:230)
at rq.a(SourceFile:175)
at rq.doInBackground(SourceFile:147)
at javax.swing.SwingWorker$1.call(SwingWorker.java:278)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at javax.swing.SwingWorker.run(SwingWorker.java:317)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
============= end nested exception, level (1) ===========
------------ END SHUTDOWN ERROR STACK -------------

I added some more code to RAFContainer to record the stack of "who" was closing the container. And just now the error occurred for me, here's the stack of the culprit who closed the container:

java.lang.Error: Closed via closeContainer
at org.apache.derby.impl.store.raw.data.RAFContainer.closeContainer(RAFContainer.java:203)
at org.apache.derby.impl.store.raw.data.RAFContainer4.closeContainer(RAFContainer4.java:183)
at org.apache.derby.impl.store.raw.data.FileContainer.clearIdentity(FileContainer.java:473)
at org.apache.derby.impl.services.cache.ConcurrentCache.evictEntry(ConcurrentCache.java:188)
at org.apache.derby.impl.services.cache.ClockPolicy.rotateClock(ClockPolicy.java:448)
at org.apache.derby.impl.services.cache.ClockPolicy.insertEntry(ClockPolicy.java:176)
at org.apache.derby.impl.services.cache.ConcurrentCache.insertIntoFreeSlot(ConcurrentCache.java:208)
at org.apache.derby.impl.services.cache.ConcurrentCache.find(ConcurrentCache.java:284)
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(BaseDataFileFactory.java:629)
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openDroppedContainer(BaseDataFileFactory.java:579)
at org.apache.derby.impl.store.raw.xact.Xact.openDroppedContainer(Xact.java:1305)
at org.apache.derby.impl.store.raw.data.ContainerBasicOperation.findContainer(ContainerBasicOperation.java:145)
at org.apache.derby.impl.store.raw.data.ContainerBasicOperation.needsRedo(ContainerBasicOperation.java:213)
at org.apache.derby.impl.store.raw.log.FileLogger.redo(FileLogger.java:1395)
at org.apache.derby.impl.store.raw.log.LogToFile.recover(LogToFile.java:920)
at org.apache.derby.impl.store.raw.RawStore.boot(RawStore.java:334)
at org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1999)
at org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:291)
at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(BaseMonitor.java:553)
at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Monitor.java:427)
at org.apache.derby.impl.store.access.RAMAccessManager.boot(RAMAccessManager.java:1019)
at org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1999)
at org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:291)
at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(BaseMonitor.java:553)
at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Monitor.java:427)
at org.apache.derby.impl.db.BasicDatabase.bootStore(BasicDatabase.java:780)
at org.apache.derby.impl.db.BasicDatabase.boot(BasicDatabase.java:196)
at org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1999)
at org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:291)
at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(BaseMonitor.java:1836)
at org.apache.derby.impl.services.monitor.BaseMonitor.startProviderService(BaseMonitor.java:1702)
at org.apache.derby.impl.services.monitor.BaseMonitor.findProviderAndStartService(BaseMonitor.java:1531)
at org.apache.derby.impl.services.monitor.BaseMonitor.startPersistentService(BaseMonitor.java:1001)
at org.apache.derby.iapi.services.monitor.Monitor.startPersistentService(Monitor.java:550)
at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(EmbedConnection.java:2416)
at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:355)
at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(EmbedConnection30.java:73)
at org.apache.derby.jdbc.Driver30.getNewEmbedConnection(Driver30.java:80)
at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:238)
at org.apache.derby.jdbc.EmbeddedDataSource.getConnection(EmbeddedDataSource.java:479)
at org.apache.derby.jdbc.EmbedPooledConnection.openRealConnection(EmbedPooledConnection.java:171)
at org.apache.derby.jdbc.EmbedPooledConnection.<init>(EmbedPooledConnection.java:113)
at org.apache.derby.jdbc.Driver30.getNewPooledConnection(Driver30.java:141)
at org.apache.derby.jdbc.EmbeddedConnectionPoolDataSource.createPooledConnection(EmbeddedConnectionPoolDataSource.java:129)
at org.apache.derby.jdbc.EmbeddedConnectionPoolDataSource.getPooledConnection(EmbeddedConnectionPoolDataSource.java:75)

Trejkaz
added a comment - 16/Nov/08 22:50 I added some more code to RAFContainer to record the stack of "who" was closing the container. And just now the error occurred for me, here's the stack of the culprit who closed the container:
java.lang.Error: Closed via closeContainer
at org.apache.derby.impl.store.raw.data.RAFContainer.closeContainer(RAFContainer.java:203)
at org.apache.derby.impl.store.raw.data.RAFContainer4.closeContainer(RAFContainer4.java:183)
at org.apache.derby.impl.store.raw.data.FileContainer.clearIdentity(FileContainer.java:473)
at org.apache.derby.impl.services.cache.ConcurrentCache.evictEntry(ConcurrentCache.java:188)
at org.apache.derby.impl.services.cache.ClockPolicy.rotateClock(ClockPolicy.java:448)
at org.apache.derby.impl.services.cache.ClockPolicy.insertEntry(ClockPolicy.java:176)
at org.apache.derby.impl.services.cache.ConcurrentCache.insertIntoFreeSlot(ConcurrentCache.java:208)
at org.apache.derby.impl.services.cache.ConcurrentCache.find(ConcurrentCache.java:284)
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(BaseDataFileFactory.java:629)
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openDroppedContainer(BaseDataFileFactory.java:579)
at org.apache.derby.impl.store.raw.xact.Xact.openDroppedContainer(Xact.java:1305)
at org.apache.derby.impl.store.raw.data.ContainerBasicOperation.findContainer(ContainerBasicOperation.java:145)
at org.apache.derby.impl.store.raw.data.ContainerBasicOperation.needsRedo(ContainerBasicOperation.java:213)
at org.apache.derby.impl.store.raw.log.FileLogger.redo(FileLogger.java:1395)
at org.apache.derby.impl.store.raw.log.LogToFile.recover(LogToFile.java:920)
at org.apache.derby.impl.store.raw.RawStore.boot(RawStore.java:334)
at org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1999)
at org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:291)
at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(BaseMonitor.java:553)
at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Monitor.java:427)
at org.apache.derby.impl.store.access.RAMAccessManager.boot(RAMAccessManager.java:1019)
at org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1999)
at org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:291)
at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(BaseMonitor.java:553)
at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Monitor.java:427)
at org.apache.derby.impl.db.BasicDatabase.bootStore(BasicDatabase.java:780)
at org.apache.derby.impl.db.BasicDatabase.boot(BasicDatabase.java:196)
at org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1999)
at org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:291)
at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(BaseMonitor.java:1836)
at org.apache.derby.impl.services.monitor.BaseMonitor.startProviderService(BaseMonitor.java:1702)
at org.apache.derby.impl.services.monitor.BaseMonitor.findProviderAndStartService(BaseMonitor.java:1531)
at org.apache.derby.impl.services.monitor.BaseMonitor.startPersistentService(BaseMonitor.java:1001)
at org.apache.derby.iapi.services.monitor.Monitor.startPersistentService(Monitor.java:550)
at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(EmbedConnection.java:2416)
at org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:355)
at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(EmbedConnection30.java:73)
at org.apache.derby.jdbc.Driver30.getNewEmbedConnection(Driver30.java:80)
at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:238)
at org.apache.derby.jdbc.EmbeddedDataSource.getConnection(EmbeddedDataSource.java:479)
at org.apache.derby.jdbc.EmbedPooledConnection.openRealConnection(EmbedPooledConnection.java:171)
at org.apache.derby.jdbc.EmbedPooledConnection.<init>(EmbedPooledConnection.java:113)
at org.apache.derby.jdbc.Driver30.getNewPooledConnection(Driver30.java:141)
at org.apache.derby.jdbc.EmbeddedConnectionPoolDataSource.createPooledConnection(EmbeddedConnectionPoolDataSource.java:129)
at org.apache.derby.jdbc.EmbeddedConnectionPoolDataSource.getPooledConnection(EmbeddedConnectionPoolDataSource.java:75)

We gave up using RAFContainer4 due to the intermittent channel closed exceptions. We modified the derby code to use the regular RAFContainer which just uses input/output streams. Since that change, we haven't had any issues, and performance was pretty much the same anyway.

Its not easy to reproduce... but there must be some subtle issue with the way channels are used / closed under Win32.

David Sitsky
added a comment - 07/May/09 04:46 We gave up using RAFContainer4 due to the intermittent channel closed exceptions. We modified the derby code to use the regular RAFContainer which just uses input/output streams. Since that change, we haven't had any issues, and performance was pretty much the same anyway.
Its not easy to reproduce... but there must be some subtle issue with the way channels are used / closed under Win32.

If the stack trace above really is the culprit, it means that we evict an item from the container cache while it's still in use. Since items are evicted, I assume the number of tables/indexes in the query mix is fairly high. Default container cache size is 100.

Knut Anders Hatlen
added a comment - 03/Jul/09 16:59 Triaged for 10.5.2.
NPE on the same code location has also been reported in DERBY-3524 and DERBY-4167 .
If the stack trace above really is the culprit, it means that we evict an item from the container cache while it's still in use. Since items are evicted, I assume the number of tables/indexes in the query mix is fairly high. Default container cache size is 100.

Kristian Waagan
added a comment - 23/Feb/10 09:00 Hi Matthew,
I see that you're hitting the NPE during connect, and have a few questions:
are you running embedded only?
do you have multiple threads accessing the database, specifically shutting down and booting/connecting "at the same time"?
when you hit the problem, did it persist consistently (i.e. it happened on every subsequent connection attempt)?
Thanks,

Knut Anders Hatlen
added a comment - 23/Feb/10 09:32 > So does "Triaged for 10.5.2." mean it is going to get fixed in that version - and any ideas when that is likely to be?
It just means that we reviewed it in the 10.5.2 bug triage and made sure that the bug meta data (component, priority, urgency, affect version, etc...) had sensible values.

Lily Wei
added a comment - 23/Feb/10 18:48 While modify system/mailjdbc test, I encounter this bug. Please see the mailsdb_derby.log and https://issues.apache.org/jira/browse/DERBY-4166?focusedCommentId=12735394&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12735394 .
After running the test for three hours, I got exception like:
------------ BEGIN SHUTDOWN ERROR STACK -------------
ERROR XSDG0: Page Page(4,Container(0, 144)) could not be read from disk.^M
at org.apache.derby.iapi.error.StandardException.newException(StandardEx
ception.java:336)
at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.j
ava:695)
at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(CachedPag
e.java:190)
...
============= begin nested exception, level (1) ===========
java.nio.channels.ClosedChannelException
at sun.nio.ch.FileChannelImpl.ensureOpen(Unknown Source)
at sun.nio.ch.FileChannelImpl.read(Unknown Source)
at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(RAFContai
ner4.java:482)
at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage0(RAFConta
iner4.java:244)
at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(RAFContai
ner4.java:214)
To reproduce this issue, please apply the patch (mailjdbc_patch.diff) for mailjdbc test and run it with embedded mode for more than 3 hours.

Kathey Marsden
added a comment - 24/Feb/10 19:03 Hi Lily,
Mike mentioned to me today that the ClosedChannelException might just be the result of an interrupted thread in the test. Do we still do a thread interrupt anywhere in mailjdbc?

Hi There:
Yes, the mailjdbc Refresh thread will be interrupted by Browse thread if it is finishing the doWork() method and the thread is sleeping. I also see XSDG9: Derby thread received an interrupt during a disk I/O operation.
ERROR XSDG9: Derby thread received an interrupt during a disk I/O operation, ple
ase check your application for the source of the interrupt.
at org.apache.derby.iapi.error.StandardException.newException(StandardEx
ception.java:279)
at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(RAFContai
ner4.java:489)
at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage0(RAFConta
iner4.java:244)

The derby.log and mailjdbc_patch022410.diff is attached.
The mailjdbc_patch022410.diff include take out select from Refresh.java, change the browse mail to read message and read attach message. Add more sleep between thread to avoid deadlock.

Lily Wei
added a comment - 24/Feb/10 21:16 Hi There:
Yes, the mailjdbc Refresh thread will be interrupted by Browse thread if it is finishing the doWork() method and the thread is sleeping. I also see XSDG9: Derby thread received an interrupt during a disk I/O operation.
ERROR XSDG9: Derby thread received an interrupt during a disk I/O operation, ple
ase check your application for the source of the interrupt.
at org.apache.derby.iapi.error.StandardException.newException(StandardEx
ception.java:279)
at org.apache.derby.impl.store.raw.data.RAFContainer4.readFull(RAFContai
ner4.java:489)
at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage0(RAFConta
iner4.java:244)
The derby.log and mailjdbc_patch022410.diff is attached.
The mailjdbc_patch022410.diff include take out select from Refresh.java, change the browse mail to read message and read attach message. Add more sleep between thread to avoid deadlock.

So it must be containerCache that's null. The only code I find that sets containerCache to null is the method markCorrupt(), which is typically called when Derby's store module encounters an IOException, or some other error that Derby doesn't know how to recover from except by rebooting the database. The exception that causes markCorrupt() to be called should be re-thrown, so I would have expected that to be present in derby.log too.

Lily, it wasn't quite clear to me from your comments whether you saw a NullPointerException in addition to the ClosedChannelException, or if you only saw the ClosedChannelException. Do you remember? It looks like a ClosedChannelException at that point in the code will cause a call to markCorrupt(), which again could make other threads accessing the same database fail with a NullPointerException.

Knut Anders Hatlen
added a comment - 16/Sep/10 10:33 Judging by the line numbers in David's stack trace (from Derby 10.4.1.3), the NullPointerException is thrown on this line in BaseDataFileFactory.openContainer():
// see if the container exists
FileContainer container = (FileContainer) containerCache.find(identity);
So it must be containerCache that's null. The only code I find that sets containerCache to null is the method markCorrupt(), which is typically called when Derby's store module encounters an IOException, or some other error that Derby doesn't know how to recover from except by rebooting the database. The exception that causes markCorrupt() to be called should be re-thrown, so I would have expected that to be present in derby.log too.
Lily, it wasn't quite clear to me from your comments whether you saw a NullPointerException in addition to the ClosedChannelException, or if you only saw the ClosedChannelException. Do you remember? It looks like a ClosedChannelException at that point in the code will cause a call to markCorrupt(), which again could make other threads accessing the same database fail with a NullPointerException.

Hi Knut:
Thank you so much for looking at the issue. This will be a good one to get fix. I remember I saw a lot of strange behavior after the ClosedChannelException, NullPointerException can be one of them. I did not capture other exception because I thought we might be too late to fix other exception. I was assuming the root cause is the ClosedChannelException. As you can see in the log, we got more than one ClosedChannelException and the tests is calling a lot of interrupt which can call makeCorrupt(). Is there something such as narrow the test case done we can do to help? Thanks!

Lily Wei
added a comment - 16/Sep/10 16:20 Hi Knut:
Thank you so much for looking at the issue. This will be a good one to get fix. I remember I saw a lot of strange behavior after the ClosedChannelException, NullPointerException can be one of them. I did not capture other exception because I thought we might be too late to fix other exception. I was assuming the root cause is the ClosedChannelException. As you can see in the log, we got more than one ClosedChannelException and the tests is calling a lot of interrupt which can call makeCorrupt(). Is there something such as narrow the test case done we can do to help? Thanks!

Lily Wei
added a comment - 20/Sep/10 00:01 When testing 10.6.2.0 with mailjdbc test, I experienced this issue again. Attach derby.log for reference. I reproduced it by running mailjdbc more than 30 hours with embedded engine.

Kathey Marsden
added a comment - 20/Sep/10 18:49 Lily, I don't actually see a NullPointerException in the derby.log. It looks like the interrupts in the test cause the ClosedChannelExceptions but I seem to be missing the NullPointerException.

Hi Kathey:
Yes, there is no NullPointerException in the derby.log. I am seeing XSDG0: Page Page(626,Container(0, 1136)) could not be read from disk and CloseChannelException after running 30 hours of mailjdbc test. These can very well cause by interrupts in the test. If the CloseChannelException I am seeing from mailjdbc test does not relate to NullPointerException while evaluating an expression. I am sorry for the confusion. Thanks Lily.

Lily Wei
added a comment - 20/Sep/10 19:39 Hi Kathey:
Yes, there is no NullPointerException in the derby.log. I am seeing XSDG0: Page Page(626,Container(0, 1136)) could not be read from disk and CloseChannelException after running 30 hours of mailjdbc test. These can very well cause by interrupts in the test. If the CloseChannelException I am seeing from mailjdbc test does not relate to NullPointerException while evaluating an expression. I am sorry for the confusion. Thanks Lily.

triage for derby 10.9. I think we should close this issue as can not reproduce. From analysis of the the stack traces I believe the original user reported
problem is likely fixed by the interrupt work done in 10.8, in the linked issue DERBY-4741.

Mike Matrigali
added a comment - 21/Feb/12 08:05 triage for derby 10.9. I think we should close this issue as can not reproduce. From analysis of the the stack traces I believe the original user reported
problem is likely fixed by the interrupt work done in 10.8, in the linked issue DERBY-4741 .

Exact same stack trace as I last listed, occurred this morning and 2 days ago. Restart of service fixed the issue both times. Regarding location of database, it is on local filesystem on a virtual machine.

Matthew Cooper
added a comment - 23/Feb/12 01:28 Problem is not fixed in 10.8.1.2.
Exact same stack trace as I last listed, occurred this morning and 2 days ago. Restart of service fixed the issue both times. Regarding location of database, it is on local filesystem on a virtual machine.
To answer Kristian's questions (sorry, I didn't receive mail notifications for comments):
are you running embedded only?
Embedded but running network server on localhost for other processes to access
do you have multiple threads accessing the database, specifically shutting down and booting/connecting "at the same time"?
There are multiple threads accessing the database, but the database is 'booted' before these threads start.
when you hit the problem, did it persist consistently (i.e. it happened on every subsequent connection attempt)?
Yes, until the process is restarted.
To repeat, this is the stack trace we are getting.
23-Feb-2012 04:36:07.788 WARN 2031264-17 com.eventzero.occurrence.service.rule.RuleService | Unexpected exception in rule service
java.lang.RuntimeException: Could not create connection for [ConnectionPool@2fd09539 driverClassName=org.apache.derby.jdbc.EmbeddedDriver username=CORLOG]
at com.eventzero.util.jdbc.ConnectionPool.createConnection(ConnectionPool.java:205) [common-e7817d5958e473be2275283ab647b8ca.jar:v3.6.6_29]
at com.eventzero.util.jdbc.ConnectionPool.take(ConnectionPool.java:125) [common-e7817d5958e473be2275283ab647b8ca.jar:v3.6.6_29]
at com.eventzero.occurrence.jdbc.AbstractDao.getConnection(AbstractDao.java:169) [corlog-core-64e317f6de56715c87e74d9a225d7c20.jar:v3.6.6_29]
at com.eventzero.summary.dao.SummaryDAO.query(SummaryDAO.java:359) [occurrence-summary-engine-efdda245967ddcae7aa0b6c8902c36c5.jar:v3.6.6_29]
at com.eventzero.summary.query.Query.getEvents(Query.java:60) [occurrence-summary-engine-efdda245967ddcae7aa0b6c8902c36c5.jar:v3.6.6_29]
at com.eventzero.summary.Rule.query(Rule.java:394) [occurrence-summary-engine-efdda245967ddcae7aa0b6c8902c36c5.jar:v3.6.6_29]
at com.eventzero.summary.RuleManager.query(RuleManager.java:170) [occurrence-summary-engine-efdda245967ddcae7aa0b6c8902c36c5.jar:v3.6.6_29]
at com.eventzero.summary.SummaryEngine.query(SummaryEngine.java:306) [occurrence-summary-engine-efdda245967ddcae7aa0b6c8902c36c5.jar:v3.6.6_29]
at com.eventzero.occurrence.service.rule.RuleService$SummaryEngineQueryRuleController.start(RuleService.java:577) [corlog-7b41983585314582e052f288c5a31a6f.jar:v3.6.6_29]
at com.eventzero.occurrence.service.rule.RuleService$Rule$1.run(RuleService.java:851) [corlog-7b41983585314582e052f288c5a31a6f.jar:v3.6.6_29]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [na:1.6.0_17]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [na:1.6.0_17]
at java.lang.Thread.run(Thread.java:619) [na:1.6.0_17]
Caused by: java.sql.SQLException: Java exception: ': java.lang.NullPointerException'.
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source) [derby-10.8.1.2.jar:na]
at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source) [derby-10.8.1.2.jar:na]
at org.apache.derby.impl.jdbc.Util.javaException(Unknown Source) [derby-10.8.1.2.jar:na]
at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source) [derby-10.8.1.2.jar:na]
at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source) [derby-10.8.1.2.jar:na]
at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source) [derby-10.8.1.2.jar:na]
at org.apache.derby.impl.jdbc.EmbedConnection.<init>(Unknown Source) [derby-10.8.1.2.jar:na]
at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(Unknown Source) [derby-10.8.1.2.jar:na]
at org.apache.derby.impl.jdbc.EmbedConnection40.<init>(Unknown Source) [derby-10.8.1.2.jar:na]
at org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Unknown Source) [derby-10.8.1.2.jar:na]
at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source) [derby-10.8.1.2.jar:na]
at org.apache.derby.jdbc.AutoloadedDriver.connect(Unknown Source) [derby-10.8.1.2.jar:na]
at java.sql.DriverManager.getConnection(DriverManager.java:582) [na:1.6.0_17]
at java.sql.DriverManager.getConnection(DriverManager.java:185) [na:1.6.0_17]
at com.eventzero.util.jdbc.ConnectionPool.createConnection(ConnectionPool.java:201) [common-e7817d5958e473be2275283ab647b8ca.jar:v3.6.6_29]
... 12 common frames omitted
Caused by: org.apache.derby.impl.jdbc.EmbedSQLException: Java exception: ': java.lang.NullPointerException'.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source) [derby-10.8.1.2.jar:na]
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source) [derby-10.8.1.2.jar:na]
... 27 common frames omitted
Caused by: java.lang.NullPointerException: null
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(Unknown Source) [derby-10.8.1.2.jar:na]
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.openContainer(Unknown Source) [derby-10.8.1.2.jar:na]
at org.apache.derby.impl.store.raw.xact.Xact.openContainer(Unknown Source) [derby-10.8.1.2.jar:na]
at org.apache.derby.impl.store.access.conglomerate.OpenConglomerate.init(Unknown Source) [derby-10.8.1.2.jar:na]
at org.apache.derby.impl.store.access.heap.Heap.open(Unknown Source) [derby-10.8.1.2.jar:na]
at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(Unknown Source) [derby-10.8.1.2.jar:na]
at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(Unknown Source) [derby-10.8.1.2.jar:na]
at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndexMinion(Unknown Source) [derby-10.8.1.2.jar:na]
at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndex(Unknown Source) [derby-10.8.1.2.jar:na]
at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.locateSchemaRow(Unknown Source) [derby-10.8.1.2.jar:na]
at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getSchemaDescriptor(Unknown Source) [derby-10.8.1.2.jar:na]
at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.initDefaultSchemaDescriptor(Unknown Source) [derby-10.8.1.2.jar:na]
at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.initialize(Unknown Source) [derby-10.8.1.2.jar:na]
at org.apache.derby.impl.db.BasicDatabase.setupConnection(Unknown Source) [derby-10.8.1.2.jar:na]
at org.apache.derby.impl.jdbc.TransactionResourceImpl.startTransaction(Unknown Source) [derby-10.8.1.2.jar:na]
at org.apache.derby.impl.jdbc.EmbedConnection.checkUserIsNotARole(Unknown Source) [derby-10.8.1.2.jar:na]
at org.apache.derby.impl.jdbc.EmbedConnection.checkUserCredentials(Unknown Source) [derby-10.8.1.2.jar:na]
... 21 common frames omitted