Now since citadel started using LibEv the possible number of simultaneusly open file descriptors grew enormously, it is like sitting down backwards onto a circular saw, and afterwards find out which tooth bite you.

You may already have heard about execinfo which enables you to fetch stack traces from within your running program to aid online debugging; certain library errormessages from citadel and webcit are able to show where they are called from this way.

So we now need to find out, whats going on with filedescriptors, who opens and closes them. strace is a nice tool for this job, however it doesn't utilize execinfo, so you only see FDs opening, closing, reading & writing, but no info from where within your program this was done.

This tiny .so can be preloaded via

LD_PRELOAD=libbacktrace.so

and will output many open / close / connect calls into a logfile so you can see what was going on. Its not all complete, but it already did its job and found my bug - so I'd like to share it with you: