Hangs after threads are aborted

Details

Description

In our production system thread occasionally get aborted to prevent long-running tasks from using resources when the user is no longer interested in the results. We are seeing these kinds of errors in the log:

Also, we are seeing hangs in MessageStreamListener.cs, line 400:
while ((line = reader.ReadLine()) != null)

This will occasionally hang indefinitely waiting for data to come in to the stream when there is no more data coming in.
I have created a reproduction application that creates a bunch of threads in an "abortable" thread pool (which just allows you to abort the threads, instead of being a black box). Starting the application will print a "." for every thread started, a "!" for every thread stopped and an "x" for every thread aborted. The thread itself just does a simple set/get. It typically hangs after about 15-20 seconds, on this ReadLine statement.

You can find the repro attached to this ticket.

This happens with Couchbase server 1.8.1 and the latest client code from Github.

Please find the attached program HangRepro-NCBC-111.zip that I used to reproduce the scenario.
The output is "output-with-using-statement.txt" (attached herewith)
The program hangs after 15-20 seconds.

Now if we change the portion of this code so as to remove the using statement and avoid recreation of client everytime a thread is created, it looks like this:

while (true)
{
Console.Write(".");
var client = new CouchbaseClient(section);
// Start a thread that just does a simple add/get and then stops
var wi = AbortableThreadPool.QueueUserWorkItem(_ =>

With the above change in code, when I run the program again, it never hangs, I kept it running for 2-3 minutes, it was working fine.
The output is "output-without-using-statement.txt" (attached herewith)

Saakshi Manocha
added a comment - 21/May/13 3:09 AM Cluster version - 2.0.1-170
Client - latest from Github
Please find the attached program HangRepro- NCBC-111 .zip that I used to reproduce the scenario.
The output is "output-with-using-statement.txt" (attached herewith)
The program hangs after 15-20 seconds.
Now if we change the portion of this code so as to remove the using statement and avoid recreation of client everytime a thread is created, it looks like this:
while (true)
{
Console.Write(".");
var client = new CouchbaseClient(section);
// Start a thread that just does a simple add/get and then stops
var wi = AbortableThreadPool.QueueUserWorkItem(_ =>
{
client.Store(StoreMode.Add, "somekey", "somevalue");
var someValue = client.Get<string>("somekey");
if (someValue != "somevalue") throw new InvalidOperationException();
Console.Write("!");
}
);
// Maybe kill the thread
if (rnd.NextDouble() < 0.75) AbortableThreadPool.Cancel(wi, true);
// Wait a bit
Thread.Sleep((int)Math.Floor(rnd.NextDouble() * 10.0));
// Maybe kill the thread
if (rnd.NextDouble() < 0.75) AbortableThreadPool.Cancel(wi, true);
}
With the above change in code, when I run the program again, it never hangs, I kept it running for 2-3 minutes, it was working fine.
The output is "output-without-using-statement.txt" (attached herewith)