As for adding the trace messages to threads.xs, not sure how to go about that. I searched for it in the perl installation folder for it but it was not there. Is it part of the threads.dll?

The easiest way would be to grab the latest version of threads from cpan, then build and install it manually. (Once you've checked you can still reproduce the error with the newly built version), you can then modify threads.xs, re-build and install.

Do you think any harm would be caused from doing something like my $val = eval{$join->join()}; as just a way to prevent the server from crashing.

If you are not doing anything with the return value from the join, it would probably be an okay temporary workaround. If you're only joining teh thread to make it go away, and it is going away of its own accord, in one sense, job done.

But in the long term, whatever the cause might just be a symptom of a deeper issue in your code; or in the threads module; or perl itself. As you have a recreatable scenario, it would be silly not to use it to help track down the underlying issue.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

Hey thanks for explanation of how to do that. I got what you asked for, and here is the link to a text file of the output from the start to finish https://dl.dropbox.com/u/19686501/serverOutput.txt None of that info is particularly helpful to me at least but if you see something I don't, I'm all ears. Thanks for your patience in me getting back to you. Took a while to recreate this time.

Just prior to a join attempt, 'another thread' with the same OS thread handle 2d00, this time does not have an OS thread id, which indicates that the thread handle:2d00 is indeed an invalid handle as the system reports.

What that indicates is that either:

The OS is reusing the same OS thread handle -- which whilst possible seems unlikely.

Or this; threads->list(threads::joinable) is returning the handle of an already joined thread. Which also seems unlikely, but could happen if the (Perl) internal linked list got corrupted some how.

The next thing I would try is adding a similar trace line at the end of S_ithread_create(), something like:

The idea is to isolate whether -- when the error occurs -- the invalid handle is to a thread that has already been freed -- in which case the bug is in threads::list() -- or to a thread that has not yet been freed -- in which case it would mean an OS error of some kind; perhaps resource constraint;

I breifly looked at trying to run your server here and trying to re-create the failure. Whilst the server runs and accepts connections from a telnet seesion, it won't accept input from it because (my) telnet sends character by character and it is expecting entire commands wrapped in your (incredibly complicated) comms protcol.

There is no way I am going to be able to reverse engineer a client that can talk that protocol.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

Alrighty, took a bit longer to recreate this time for whatever reason. I made the changes you suggested, but in the line printf( "ITFREE: thread handle:%x thread-id: %dx\n", thread->handle, GetThreadId( thread->handle ) ); i changed thread->handle to just 'handle' since it looks they already freed that pointer at that point.
I attached the log of this run below in the files called serverOutput2.txt, it exited with a different error this time,

Which again appears to be related to the thread handle.
Sorry about the complicated protocol. People I work with did not wish to go through the trouble of having much back and forth communication between the client/server, and rather just send command once, receive response, and still needed a way to transfer a 400MB file. So this is what we (I) came up with (you should have seen the earlier version). I've attached the client (rx.pl) as well as the test script i used to recreate this issue. (i also included the server, rxd.pl, with all other commands besides exec stripped out except for EXEC to shorten the code. And included the exact threads.xs used to compile the threads module)
Thanks again for the help.
Files:https://dl.dropbox.com/u/19686501/perlmonk.zip

When putting a smiley right before a closing parenthesis, do you:

Use two parentheses: (Like this: :) )
Use one parenthesis: (Like this: :)
Reverse direction of the smiley: (Like this: (: )
Use angle/square brackets instead of parentheses
Use C-style commenting to set the smiley off from the closing parenthesis
Make the smiley a dunce: (:>
I disapprove of emoticons
Other