"Broken Pipe" and "client is busy or timeout" errors

"Broken Pipe" and "client is busy or timeout" errors

I am facing two kind of excpetion actually with grizlly comet support. When the handler try to write in the response, I sometimes have "Broken Pipe" or "client is busy or timeout".

I am using a concurrent comet handler because I use comet in a streaming mode. I have just modified the DefaultConcurrentCometHandler in order to give a limite a message send by one thread. The comet handler keep a reference on the httpresponse from the comet servlet when the user register.

I am using grizzly 1.0.39 in glassfish v2.

I have read some pieces of code and actually I see that the second exception is due to a timeout on a selector.select() that took too much time. Just for my understanding, did a call to the select method really send the data to the remote client ?

So, in order to try to fix my issues I have to know if they are due to grizzly internal logic (or my logic around grizzly) or due to netwrok settings that are not properly set.

Does anyone can help me ?

Thanks.

William.

Stack Trace :

ClientAbortException: java.io.IOException: Broken pipe
at org.apache.coyote.tomcat5.OutputBuffer.doFlush(OutputBuffer.java:386)
at org.apache.coyote.tomcat5.OutputBuffer.flush(OutputBuffer.java:352)
at org.apache.coyote.tomcat5.CoyoteOutputStream.flush(CoyoteOutputStream.java:176)
at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:278)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122)
at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212)
......
at com.sun.enterprise.web.connector.grizzly.comet.DefaultNotificationHandler.notify(DefaultNotificationHandler.java:149)
at com.sun.enterprise.web.connector.grizzly.comet.CometContext.notify(CometContext.java:553)
......
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
at sun.nio.ch.IOUtil.write(IOUtil.java:71)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
at com.sun.enterprise.web.connector.grizzly.OutputWriter.flushChannel(OutputWriter.java:92)
at com.sun.enterprise.web.connector.grizzly.OutputWriter.flushChannel(OutputWriter.java:65)
at com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flushChannel(SocketChannelOutputBuffer.java:170)
at com.sun.enterprise.web.connector.grizzly.async.AsynchronousOutputBuffer.flushChannel(AsynchronousOutputBuffer.java:81)
at com.sun.enterprise.web.connector.grizzly.SocketChannelOutputBuffer.flushBuffer(SocketChannelOutputBuffer.java:203)
at com.sun.enterprise.web.connector.grizzly.async.AsynchronousOutputBuffer.flushBuffer(AsynchronousOutputBuffer.java:113)

Re: "Broken Pipe" and "client is busy or timeout" errors

Administrator

Hi William,

pls. try to increase the keep-alive timeout-in-seconds attr [1] in the
GF domain.xml file, make it much bigger and check if you still have
exception coming.

> I am using a concurrent comet handler because I use comet in a streaming
> mode. I have just modified the DefaultConcurrentCometHandler in order to
> give a limite a message send by one thread. The comet handler keep a
> reference on the httpresponse from the comet servlet when the user register.
>
> I am using grizzly 1.0.39 in glassfish v2.
>
> I have read some pieces of code and actually I see that the second exception
> is due to a timeout on a selector.select() that took too much time. Just for
> my understanding, did a call to the select method really send the data to
> the remote client ?

channel.write() sends them, selector.select() just listens for
channel(s) I/O events and wakes up if any is available for processing.

> So, in order to try to fix my issues I have to know if they are due to
> grizzly internal logic (or my logic around grizzly) or due to netwrok
> settings that are not properly set.
Difficult to say, let's try to increase timeout and see.
If it doesn't work, can you pls. provide a simple testcase so we can
take a look at?

Re: "Broken Pipe" and "client is busy or timeout" errors

ok I will try to increase the timeout.... This change should fix the 2 errors ("Broken Pipe" or "client is busy or timeout") or only one ? Can you explain why it should help ?

if channel.write send the data, why do we have to call selector.select() ? After som check on client side, it seams that the connection is still alive but selector.select() is returning 0.

it is difficult to provide a test case because it happened sometimes, mainly when the load increase. Our use of comet is simple, we have implemented a comet handler inspired from ConcurrentCometHandler and we write in the httpResponse each time we have to write an event.

We implement our own thread pool in order to handler priority and use several comet context in the same comet engine.

Re: "Broken Pipe" and "client is busy or timeout" errors

On 03/28/2011 02:45 PM, wild wrote:
> ok I will try to increase the timeout.... This change should fix the 2 errors
> ("Broken Pipe" or "client is busy or timeout") or only one ? Can you explain
> why it should help ?
Primarily "broken pipe", not sure where the 2nd exception comes from yet.

> if channel.write send the data, why do we have to call selector.select() ?
> After som check on client side, it seams that the connection is still alive
> but selector.select() is returning 0.
It depends which selector.select() you mean, the one in
OutputWriter.flushChannel()?
It's getting called when we can not flush the entire buffer, because the
underlying socket buffer is overloaded. (most probably client is not
able to read data as fast as server sends it). So we just wait until
client releases some space in the TCP buffer and let server write more data.

You can increase the write timeout as well by setting jvm-option
-Dcom.sun.enterprise.web.connector.grizzly.writeTimeout=XXX
in the domain.xml, the value is in milliseconds, by default it's equal
to 30000 (30 seconds).

> it is difficult to provide a test case because it happened sometimes, mainly
> when the load increase. Our use of comet is simple, we have implemented a
> comet handler inspired from ConcurrentCometHandler and we write in the
> httpResponse each time we have to write an event.
>
> We implement our own thread pool in order to handler priority and use
> several comet context in the same comet engine.
I see. Let's check what timeouts increasing will give us.

Re: "Broken Pipe" and "client is busy or timeout" errors

One more thing, when I display grizzly configuration in server.log file, it
> shows
>
> socket-keep-alive-enabled: false
>
> In order to make keep alive timeout active, should it be true ?
>
It's a bit different, this one is socket keep-alive and we're trying to
increase *HTTP* keep-live timeout :)

> I will try to increase socket buffer. Do you think it can help ?
From what you're reporting, it looks like bottleneck occurs when server
writes response to client, which should mean client is not able to
process server response as fast as server sends it.
If that's the case - you can try to increase server write timeout i
mentioned in prev. post.

The increasing of socket buffer may help to delay the problem, but not
sure if it would resolve it. Depends on your concrete usecase.

So If a increase the server timeout, i will keep a thread to many times and enqueue too many messages. If the client is too slow to read messages, then i think the only solution is ti reduce the number of messages send.