There are cases in NSConnection when the use of NSRecursiveLocks between objects can lead to 2 threads deadlocking.

The case I have seen is in gnustep-base v1.24.4.

The NSConnection method locateLocalTarget: locks the following objects in this order

IrefGate for self

connection_table_gate

IRefGate for another NSConnection

In addition, the invalidate method locks objects in this order

IrefGate

connection_table_gate

I have seen 1 lockup scenario, and fear for another occuring.

Scenario:

NSConnection 'A' has its locateLocalTarget: methods called. It locks the local IrefGate and proceeds into the method. Later in the method it locks connection_table_gate and begins to scan the members of the connection_table.

Another NSConnection 'B' has invalidate called on another thread. It locks the local IrefGate and proceeds into the method. It later attempts to lock connection_table_gate but cannot as the thread running NSConnection 'A' has it. The thread blocks.

Back on the thread for NSConnection 'A', while enumerating connection_table it attempts to lock NSConnection 'B' IrefGate. It cannot, because the thread running NSConnection 'B' has it. So it locks.

The threads running NSConnections A and B are now deadlocked.

I am also concerned that the same deadlock can happen if both Connection A and B enter locateLocalTarget at the same time where one connection gets its local IrefGate and connection_table_gate and the other gets only its local IRefGate.

I don't have enough context on the software to recommend a solution, although I have 'tinkered' with something that seems to help (still testing).

For information, in my test application it takes about 8-12 hours of continuous run before this condition occurs.