In a test setup, we have domain "wooga" with a samba-based PDC "tserv". This
domain is trusting the domain "testdomain" with the NT4 basesd PDC "nt4server"
smbd and nmbd without winbindd do run fine. when winbindd is started, the
following sequence deadlocks smbd and winbindd:
1/ samba is waiting for a connection
2/ winbindd is starting and connecting to the local samba server. Before
it connects, it successfully acquires a mutex with
secrets_named_mutex ( "TSERV", WINBIND_SERVER_MUTEX_WAIT_TIME ), which
calls tdb_lock ( "secrets.tdb", BUCKET(tdb_hash(&"TSERV")), F_WRLCK).
This leads to the following syscall:
fcntl64(<fd of secrets.tdb>, F_SETLKW64, {type=F_WRLCK, whence=SEEK_SET,
start=280, len=1}, 0xbfffea00) = 0
3/ when winbindd connects to samba, smbd starts a new process and
initializes LDAP. During this initialization, it retrieves the domain
SID from LDAP and after that from the secrets.tdb, to compare both
values (line 2360 of passdb/pdb_ldap.c). Retrieving the domain SID from
LDAP involves locking the key SECRETS/SID/WOOGA in secrets.tdb. This
also leads to the following syscall:
fcntl64(<fd of secrets.tdb>, F_SETLKW64, {type=F_RDLCK, whence=SEEK_SET,
start=280, len=1}
4/ the last call locks smbd until winbindd is giving up and releasing
its own lock.
Note, that this does not happen, when "tserv" is renamed to "tests" or
"testserver".

Jerry, before you jump on that one: I'm currently in the process of writing
a mutex implementation that creats a directory locks/mutex/, creates lock
files in there and fcntl locks that file.
I've discussed it with abartlet on irc, he wants to do simple fcntl locks
on a hash function hashing to 31 bits.
Comment?
Volker

Created attachment 367[details]
mutex based on lock files
Ok, this is a (very) preliminary patch that implements a different
mutex scheme based on indiviual lock files. It survives its own
torture test as expected. I did not do any performance measurements yet.
Volker