Description

Sometime it shows Broken pipe exception in error log. After that acceptor seems to be dead. Exception in logged many times (huge logs up to 30gb and more). The only way to remove bug is to restart jetty.

java.io.IOException: Broken pipe
at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:256)
at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:175)
at org.mortbay.io.nio.SelectorManager$SelectSet.wakeup(SelectorManager.java:831)
at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:709)
at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

Activity

could you possibly try 6.1.26, as we made some changes to the selector manager that could improve handling of closing connections (JETTY-1296, JETTY-748, JETTY-1264). While none of these are directly related to a Broken Pipe, they may improve the selectors ability to recover from such and error.

Greg Wilkins
added a comment - 06/Mar/11 7:07 PM
Hi,
could you possibly try 6.1.26, as we made some changes to the selector manager that could improve handling of closing connections ( JETTY-1296 , JETTY-748 , JETTY-1264 ). While none of these are directly related to a Broken Pipe, they may improve the selectors ability to recover from such and error.
How frequently does this happen?
Interestingly, Glassfish/Grizzly has had a similar issue reported: http://java.net/jira/browse/GRIZZLY-813 , so we may have a look at their fix to see if this is an NIO issue.

Greg Wilkins
added a comment - 06/Mar/11 7:30 PM The Grizzly fix was simply to catch an additional IOException and the cancel the key or close the exception.
6.1.26 has already improved the exception handling around that area, so this may be fixed.
If you can reproduce with 6.1.26, then we should similarly be able to cancel the key and re-register.

Viktor Kuzmin
added a comment - 09/Mar/11 1:55 AM I'm running 6.1.26 currently. It is reproduceable. BUT it seems that bug is reproduceable only if I set acceptors to 2, can't reproduce bug with acceptors set to 4.

I've had another report about this issue. I will try to add some explicit handling into the server for it. As it appears the selector is dead or dying, I think the solution will have to remake a new selector (as we've done for other JVM bugs). Question is, will we lose existing connections?

Greg Wilkins
added a comment - 27/Mar/11 5:53 PM I've had another report about this issue. I will try to add some explicit handling into the server for it. As it appears the selector is dead or dying, I think the solution will have to remake a new selector (as we've done for other JVM bugs). Question is, will we lose existing connections?
Also I wonder why we've not had this reported on jetty-7?

I'm looking and better handling IOException in our wakeup code, but the problem is that IOException is not actually thrown by selector.wakeup. Can you tell from the log if the exception was caught and logged by jetty, or was it printed out by the JVM library itself?

Greg Wilkins
added a comment - 28/Mar/11 1:15 AM
I'm looking and better handling IOException in our wakeup code, but the problem is that IOException is not actually thrown by selector.wakeup. Can you tell from the log if the exception was caught and logged by jetty, or was it printed out by the JVM library itself?

It was caught and logged by jetty.
Also we have migrated to jetty-7 already and it seems that there is no such problem with it... So currently will be not able to reproduce/test this bug... Jetty-7 is running for four days without problems already.

Viktor Kuzmin
added a comment - 28/Mar/11 6:06 AM It was caught and logged by jetty.
Also we have migrated to jetty-7 already and it seems that there is no such problem with it... So currently will be not able to reproduce/test this bug... Jetty-7 is running for four days without problems already.

I don't know for sure if it is a fix, but I've committed some defensive code to jetty-6 trunk. If anybody that suffers from this bug could try it out and confirm that it works, then I would do a maintenance release of jetty-6

Greg Wilkins
added a comment - 17/May/11 2:05 AM I don't know for sure if it is a fix, but I've committed some defensive code to jetty-6 trunk. If anybody that suffers from this bug could try it out and confirm that it works, then I would do a maintenance release of jetty-6

As no-one has commented since May, I'm assuming that Greg's fixes in 6.1 trunk are OK. If this is not the case, then please reopen this issue, providing stack traces etc and we will take another look at it, hopefully before 6.1.27 goes out the door.

Jan Bartel
added a comment - 08/Nov/11 9:53 PM As no-one has commented since May, I'm assuming that Greg's fixes in 6.1 trunk are OK. If this is not the case, then please reopen this issue, providing stack traces etc and we will take another look at it, hopefully before 6.1.27 goes out the door.
Jan