One common task we must perform when debugging MySQL problems is, unsurprisingly, attaching a debugger to mysqld. While this is not something one usually does in a production MySQL, gathering stack traces can be a useful last action when the database is hung, and your next step would be to restart it. This way, we may end up with valuable information to diagnose why MySQL was hung in the first place, either for our own analysis or a useful bug report.

Before we can do this, we need to obtain the PID on the host system that corresponds to our container:

This failed because processes running in Docker containers have a separate namespace and therefore we can’t attach gdb to them.

Fortunately, we can leverage the command nsenter to ‘enter’ the corresponding namespace and attach gdb to mysqld. Before we can do that, however, gdb must be installed in the container. The following snippet shows how we can:

Install gdb inside the container

Attach it to mysqld to collect stack traces

Run the collected traces (now on the host OS) by pt-pmp for aggregation

As you can see, even though we have to go through some extra hoops, we were able to attach gdb to mysqld inside a container, and while my example only gathered stack traces for aggregation, this approach opens the door for us to perform any debugging activity we would typically do on a process that’s running directly on our host OS.

Conclusion

Containers offer several operational benefits including standardized deployments, better isolation and consistency between different environments. However, like anything in life, those benefits don’t come for free. If you’re a pager-carrying person who responds to incidents and is responsible for keeping systems running correctly or figuring out problems and resolving them when systems are not running correctly, the takeaway I would like you to get from this post is: prepare yourself for troubleshooting before your stack hits production. As any layer of abstraction, containers make some of our tracing and debugging work more difficult, and it is best to get familiar with how those practices work in a containerized environment without the stress of having to deal with a production problem.

About the Author

Fernando Ipar is a MySQL Principal Consultant who works remotely from his family home in Montevideo, Uruguay. Initially, Fernando began using GNU/Linux in 1998 and MySQL in 2000, and his journey led him to become active in the technical community, contributing and at times leading open source projects as well as founding the Montevideo MySQL Meetup. Before coming to Pythian, he spent seven years at Percona, where he contributed to the Percona Toolkit and TokuMX Toolkit, among other endeavors. Fernando helped scale and troubleshoot the back-ends for businesses of all sizes including financial institutions, telcos, and other IT companies while also supporting their full stack needs.

PYTHIAN®, LOVE YOUR DATA®, and ADMINISCOPE® are trademarks and registered trademarks owned by Pythian in North America and certain other countries, and are valuable assets of our company. Other brands, product and company names on this website may be trademarks or registered trademarks of Pythian or of third parties. Use of trademarks without permission is strictly prohibited.