Daemon tools stuck updating virtual devices

I assume care is taken to always defer unlock after acquiring a lock.Have you (Docker Inc as a whole, not you specifically) not got any scripts to loop over the goroutines in a running daemon and dump out what they're blocked on? I've just compiled the daemon with symbols and then loaded those symbols into a gdb attached to the docker daemon.

After the build gave up, a subsequent build started which included a script that kills any left-over containers.

It happens sometimes but I have one case where it kills a container every time. I want to make sure it is a bug before opening a ticket. Here's what happens: qba19 ~ # pgm/perf-tools/bin/tpoint syscalls:sys_enter_kill Tracing syscalls:sys_enter_kill. 67219.620193: sys_kill(pid: 3b0, sig: 0) docker-18317 [004] ....

67225.462948: sys_kill(pid: 92d, sig: 0) docker-18317 [004] ....

I can now write python scripts to traverse all the goroutines looking for particular things..I know what I'm looking for. If we grep out the callers to Lock then we find the callsites goroutines are blocked at: [email protected]:~# grep -A1 Lock | grep -v '^--$' | grep -v 'Mutex' | sort | uniq -c 1 #5 0x00000000004bc5ba in github.com/docker/docker/daemon.(*Daemon).

Do you have any links to previous tickets like this? Unfortunately there are 618 goroutines :) I've scripted their dumping, result at https://gist.github.com/aidanhs/960beaf2db1de622a2dd. Container Inspect (daemon=0xc2080691e0, job=0xc208857980, ~r1=0) at /go/src/github.com/docker/docker/daemon/inspect.go:17 1 #5 0x00000000004c861f in github.com/docker/docker/daemon.(*State).