Fixing a bug in an application, which uses a third party stack is always challenging especially if it isn't obvious at outset where the problem lies. Recently I was up against two such issues in the application that I code. The fascinating thing is that both required a single line change of code but almost twenty man-days to find out which line to change! The multithreaded client application uses the OpenSSL stack to secure the communication with the server. One issue was significant memory leak and the other a crash. Both issues did not give direct pointers to the buggy code.

Valgrind is a very good tool for debugging memory leak issues. It reported 3k bytes of leak for every connection that was made to the server. The mailing list gave hints to the solution. After about ten days of analysis and help from top-notch developer in team it turned that I was using an OpenSSL API incorrectly. The fix turned out to be calling a different method than what I was using.

The application crash problem was tougher. It couldn't make it to the targeted release of the product. The crash was occurring when the application was heavily stressed. The mailing list was of not much help. After days of investigation it turned out that the bug was in the OpenSSL code. The OpenSSL method of getting thread IDs was failing in the Linux kernel 2.6. The implementation returned the same ID for every thread. As such the stack was to be changed to fix the issue. And even this time the fix is just a change in one line of the code. Here too I should be thanking my friend who helped me in having the issue fixed and the product released. It helps in having a high quality developer in the team as very good friend of yours!