I've found slightly similar isse here: http://issues.opennms.org/browse/NMS-4631
New file handlers are being opened by Apache Mina. I am not sure who is responsible for closing it. In my testcase the connection is properly closed.

Emmanuel Lecharny
added a comment - 27/Jul/11 17:31 Ok, fix applied.
http://svn.apache.org/viewvc?rev=1151527&view=rev
Martin, many thanks for the bug report, for having persisted and reopened the issue and for having provided he code to reproduce the error.
We will try to get a release by the end of this week.

Emmanuel Lecharny
added a comment - 27/Jul/11 17:17 I do think that the comment (// DO NOT...) was added when we were using an old version of MINA, whch might have some bug.
anyway...
I have fixed the NPE by protecting the connector.dispose() call with a mutex, removing the localConnector variable.

After some debugging session with Emmanuel, we found out the following things:

A connector is initialized at first when the connection is bound.

Calling the unbind causes the server to close the session with the connection.

With the connection being closed, the sessionClose() method of the LdapNetworkConnection class is called

This method is not disposing the connector but setting it to null

Afterwards when the close() method is called on the LdapNetworkConnection, it can't dispose the connector as it has been nullified by the other method.
=> Thus, the connection is still open.

Calling the connector.dispose() method in the sessionClosed() method seem to resolve the issue, but the following comment found in that method make us think that this change needs to be double checked:
// DO NOT call connector.dispose(), it is hanging when there is no network connection

We were able to increase to loop to 10,000 iterations with a successful ending.
However, from time to time, we are hitting some NPEs, mainly due to concurrency (one method nullifying a field used in another method at the same time).

We are currently trying to find the best way to schedule the connector.dispose() method call.

Pierre-Arnaud Marcelot
added a comment - 27/Jul/11 17:06 After some debugging session with Emmanuel, we found out the following things:
A connector is initialized at first when the connection is bound.
Calling the unbind causes the server to close the session with the connection.
With the connection being closed, the sessionClose() method of the LdapNetworkConnection class is called
This method is not disposing the connector but setting it to null
Afterwards when the close() method is called on the LdapNetworkConnection, it can't dispose the connector as it has been nullified by the other method.
=> Thus, the connection is still open.
Calling the connector.dispose() method in the sessionClosed() method seem to resolve the issue, but the following comment found in that method make us think that this change needs to be double checked:
// DO NOT call connector.dispose(), it is hanging when there is no network connection
We were able to increase to loop to 10,000 iterations with a successful ending.
However, from time to time, we are hitting some NPEs, mainly due to concurrency (one method nullifying a field used in another method at the same time).
We are currently trying to find the best way to schedule the connector.dispose() method call.

Same error on my mac :
Exception in thread "main" org.apache.mina.core.RuntimeIoException: Failed to initialize.
at org.apache.mina.core.polling.AbstractPollingIoConnector.<init>(AbstractPollingIoConnector.java:183)
at org.apache.mina.core.polling.AbstractPollingIoConnector.<init>(AbstractPollingIoConnector.java:93)
at org.apache.mina.transport.socket.nio.NioSocketConnector.<init>(NioSocketConnector.java:56)
at org.apache.directory.ldap.client.api.LdapNetworkConnection.connect(LdapNetworkConnection.java:496)
at org.apache.directory.ldap.client.api.LdapNetworkConnection.bindAsync(LdapNetworkConnection.java:1287)
at org.apache.directory.ldap.client.api.LdapNetworkConnection.bind(LdapNetworkConnection.java:1144)
at org.apache.directory.ldap.client.api.LdapNetworkConnection.bind(LdapNetworkConnection.java:922)
at org.i2l.samples.LdapTestCase.main(LdapTestCase.java:20)
Caused by: java.io.IOException: Too many open files in system
at sun.nio.ch.KQueueArrayWrapper.init(Native Method)
at sun.nio.ch.KQueueArrayWrapper.<init>(KQueueArrayWrapper.java:76)
at sun.nio.ch.KQueueSelectorImpl.<init>(KQueueSelectorImpl.java:51)
at sun.nio.ch.KQueueSelectorProvider.openSelector(KQueueSelectorProvider.java:20)
at java.nio.channels.Selector.open(Selector.java:209)
at org.apache.mina.transport.socket.nio.NioSocketConnector.init(NioSocketConnector.java:131)
at org.apache.mina.core.polling.AbstractPollingIoConnector.<init>(AbstractPollingIoConnector.java:178)
... 7 more

Emmanuel Lecharny
added a comment - 27/Jul/11 15:16 Hi Martin,
yes, it seems a bit more complicated than the titlle made me think it was. (This is why picking a good tittle is important
Ok, let's investigate this issue.

2011-07-27 14:52:36,162 ERROR SimpleIoProcessorPool - Failed to create a new instance of org.apache.mina.transport.socket.nio.NioProcessor:null
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedConstructorAccessor4.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.mina.core.service.SimpleIoProcessorPool.<init>(SimpleIoProcessorPool.java:180)
at org.apache.mina.core.service.SimpleIoProcessorPool.<init>(SimpleIoProcessorPool.java:112)
at org.apache.mina.core.polling.AbstractPollingIoConnector.<init>(AbstractPollingIoConnector.java:93)
at org.apache.mina.transport.socket.nio.NioSocketConnector.<init>(NioSocketConnector.java:56)
at org.apache.directory.ldap.client.api.LdapNetworkConnection.connect(LdapNetworkConnection.java:495)
at org.apache.directory.ldap.client.api.LdapNetworkConnection.bindAsync(LdapNetworkConnection.java:1274)
at org.apache.directory.ldap.client.api.LdapNetworkConnection.bind(LdapNetworkConnection.java:1131)
at org.apache.directory.ldap.client.api.LdapNetworkConnection.bind(LdapNetworkConnection.java:907)
at mway.UserDaoApacheDirectoryLdapApiImpl.<init>(UserDaoApacheDirectoryLdapApiImpl.java:32)
at mway.UserDaoApacheDirectoryLdapApiImpl.main(UserDaoApacheDirectoryLdapApiImpl.java:18)
Caused by: org.apache.mina.core.RuntimeIoException: Failed to open a selector.
at org.apache.mina.transport.socket.nio.NioProcessor.<init>(NioProcessor.java:61)
... 13 more
Caused by: java.io.IOException: Too many open files
at sun.nio.ch.EPollArrayWrapper.epollCreate(Native Method)
at sun.nio.ch.EPollArrayWrapper.<init>(EPollArrayWrapper.java:69)
at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:52)
at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
at java.nio.channels.Selector.open(Selector.java:209)
at org.apache.mina.transport.socket.nio.NioProcessor.<init>(NioProcessor.java:59)
... 13 more
Exception in thread "main" org.apache.mina.core.RuntimeIoException: Failed to create a new instance of org.apache.mina.transport.socket.nio.NioProcessor:null
at org.apache.mina.core.service.SimpleIoProcessorPool.<init>(SimpleIoProcessorPool.java:198)
at org.apache.mina.core.service.SimpleIoProcessorPool.<init>(SimpleIoProcessorPool.java:112)
at org.apache.mina.core.polling.AbstractPollingIoConnector.<init>(AbstractPollingIoConnector.java:93)
at org.apache.mina.transport.socket.nio.NioSocketConnector.<init>(NioSocketConnector.java:56)
at org.apache.directory.ldap.client.api.LdapNetworkConnection.connect(LdapNetworkConnection.java:495)
at org.apache.directory.ldap.client.api.LdapNetworkConnection.bindAsync(LdapNetworkConnection.java:1274)
at org.apache.directory.ldap.client.api.LdapNetworkConnection.bind(LdapNetworkConnection.java:1131)
at org.apache.directory.ldap.client.api.LdapNetworkConnection.bind(LdapNetworkConnection.java:907)
at mway.UserDaoApacheDirectoryLdapApiImpl.<init>(UserDaoApacheDirectoryLdapApiImpl.java:32)
at mway.UserDaoApacheDirectoryLdapApiImpl.main(UserDaoApacheDirectoryLdapApiImpl.java:18)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedConstructorAccessor4.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.mina.core.service.SimpleIoProcessorPool.<init>(SimpleIoProcessorPool.java:180)
... 9 more
Caused by: org.apache.mina.core.RuntimeIoException: Failed to open a selector.
at org.apache.mina.transport.socket.nio.NioProcessor.<init>(NioProcessor.java:61)
... 13 more
Caused by: java.io.IOException: Too many open files
at sun.nio.ch.EPollArrayWrapper.epollCreate(Native Method)
at sun.nio.ch.EPollArrayWrapper.<init>(EPollArrayWrapper.java:69)
at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:52)
at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
at java.nio.channels.Selector.open(Selector.java:209)
at org.apache.mina.transport.socket.nio.NioProcessor.<init>(NioProcessor.java:59)
... 13 more

Again, increasing ulimit is not a solution, as this is used in a server application.

Martin Zdila
added a comment - 27/Jul/11 13:55 Even simpler testcase:
public class LdapTestcase {
public static void main(final String[] args) throws Exception {
final LdapConnectionConfig lcc = new LdapConnectionConfig();
lcc.setLdapHost("ad.example.com");
lcc.setUseSsl(true);
lcc.setLdapPort(LdapConnectionConfig.DEFAULT_LDAPS_PORT);
lcc.setTrustManagers(new NoVerificationTrustManager());
for (int i = 0; i < 1000; i++)
{
System.out.println(i);
final LdapNetworkConnection connection = new LdapNetworkConnection(lcc);
connection.bind();
connection.unBind();
connection.close();
}
}
}
Exception:
2011-07-27 14:52:36,162 ERROR SimpleIoProcessorPool - Failed to create a new instance of org.apache.mina.transport.socket.nio.NioProcessor:null
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedConstructorAccessor4.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.mina.core.service.SimpleIoProcessorPool.<init>(SimpleIoProcessorPool.java:180)
at org.apache.mina.core.service.SimpleIoProcessorPool.<init>(SimpleIoProcessorPool.java:112)
at org.apache.mina.core.polling.AbstractPollingIoConnector.<init>(AbstractPollingIoConnector.java:93)
at org.apache.mina.transport.socket.nio.NioSocketConnector.<init>(NioSocketConnector.java:56)
at org.apache.directory.ldap.client.api.LdapNetworkConnection.connect(LdapNetworkConnection.java:495)
at org.apache.directory.ldap.client.api.LdapNetworkConnection.bindAsync(LdapNetworkConnection.java:1274)
at org.apache.directory.ldap.client.api.LdapNetworkConnection.bind(LdapNetworkConnection.java:1131)
at org.apache.directory.ldap.client.api.LdapNetworkConnection.bind(LdapNetworkConnection.java:907)
at mway.UserDaoApacheDirectoryLdapApiImpl.<init>(UserDaoApacheDirectoryLdapApiImpl.java:32)
at mway.UserDaoApacheDirectoryLdapApiImpl.main(UserDaoApacheDirectoryLdapApiImpl.java:18)
Caused by: org.apache.mina.core.RuntimeIoException: Failed to open a selector.
at org.apache.mina.transport.socket.nio.NioProcessor.<init>(NioProcessor.java:61)
... 13 more
Caused by: java.io.IOException: Too many open files
at sun.nio.ch.EPollArrayWrapper.epollCreate(Native Method)
at sun.nio.ch.EPollArrayWrapper.<init>(EPollArrayWrapper.java:69)
at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:52)
at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
at java.nio.channels.Selector.open(Selector.java:209)
at org.apache.mina.transport.socket.nio.NioProcessor.<init>(NioProcessor.java:59)
... 13 more
Exception in thread "main" org.apache.mina.core.RuntimeIoException: Failed to create a new instance of org.apache.mina.transport.socket.nio.NioProcessor:null
at org.apache.mina.core.service.SimpleIoProcessorPool.<init>(SimpleIoProcessorPool.java:198)
at org.apache.mina.core.service.SimpleIoProcessorPool.<init>(SimpleIoProcessorPool.java:112)
at org.apache.mina.core.polling.AbstractPollingIoConnector.<init>(AbstractPollingIoConnector.java:93)
at org.apache.mina.transport.socket.nio.NioSocketConnector.<init>(NioSocketConnector.java:56)
at org.apache.directory.ldap.client.api.LdapNetworkConnection.connect(LdapNetworkConnection.java:495)
at org.apache.directory.ldap.client.api.LdapNetworkConnection.bindAsync(LdapNetworkConnection.java:1274)
at org.apache.directory.ldap.client.api.LdapNetworkConnection.bind(LdapNetworkConnection.java:1131)
at org.apache.directory.ldap.client.api.LdapNetworkConnection.bind(LdapNetworkConnection.java:907)
at mway.UserDaoApacheDirectoryLdapApiImpl.<init>(UserDaoApacheDirectoryLdapApiImpl.java:32)
at mway.UserDaoApacheDirectoryLdapApiImpl.main(UserDaoApacheDirectoryLdapApiImpl.java:18)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedConstructorAccessor4.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.mina.core.service.SimpleIoProcessorPool.<init>(SimpleIoProcessorPool.java:180)
... 9 more
Caused by: org.apache.mina.core.RuntimeIoException: Failed to open a selector.
at org.apache.mina.transport.socket.nio.NioProcessor.<init>(NioProcessor.java:61)
... 13 more
Caused by: java.io.IOException: Too many open files
at sun.nio.ch.EPollArrayWrapper.epollCreate(Native Method)
at sun.nio.ch.EPollArrayWrapper.<init>(EPollArrayWrapper.java:69)
at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:52)
at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
at java.nio.channels.Selector.open(Selector.java:209)
at org.apache.mina.transport.socket.nio.NioProcessor.<init>(NioProcessor.java:59)
... 13 more
Again, increasing ulimit is not a solution, as this is used in a server application.

The issue is not the low value of allowed opened files. The problem is that opened pipes never closes. See my testcase. Do you see there any issue? Do you think that it should cause so many opened pipes and anon_inodes? It is not observed just on a single environment.

If I had a program that would be opening file stream but never closing it, would you recommend me to increase ulimit too?

Martin Zdila
added a comment - 27/Jul/11 13:35 Hi Emmanuel
The issue is not the low value of allowed opened files. The problem is that opened pipes never closes. See my testcase. Do you see there any issue? Do you think that it should cause so many opened pipes and anon_inodes? It is not observed just on a single environment.
If I had a program that would be opening file stream but never closing it, would you recommend me to increase ulimit too?

Emmanuel Lecharny
added a comment - 27/Jul/11 13:23 This is not an API issue, it's an issue on your own system.
Modify its configuration to allow a higher number of files to be opened (ulimit parameter on linux)