Using New Debugging Features to Probe NGINX Internals

In recent versions of NGINX we have added a couple of useful debugging features which can help you get more information out of NGINX if something ever goes wrong. They work by using GDB to extract information from a running server. While we don’t advise that you use GDB on a running NGINX instance in production, it can be very useful in a development or testing environment.

For the features included in this blog post to work, include the --with-debug configuration option when you build NGINX. To determine whether your binary was built with the option, run the nginx-V command:

Writing the Debug Log in Memory

The first new feature is the in‑memory debug log. The NGINX debug log is very useful for digging into complex issues, but at the same time it can grow very large very quickly and eat up disk space.

In NGINX 1.7.11 we added the ability to log directly into memory using a cyclic buffer, so that debug logging does not use disk storage at all. For details, see Logging to a cyclic memory buffer in the NGINX documentation.

To enable a 32‑MB buffer for debug logging, include this error_log directive in the main context of your NGINX configuration file:

error_log memory:32m debug;

You can use GDB to extract the log from memory. To assist you with this, I’ve created a GitHub Gist with everything you need for the examples in this blog post. Download and save it as nginx.gdb, or rename it to .gdbinit in your home directory to have it load automatically.

First, run this command to display the process IDs of the NGINX worker processes:

# pgrep -f "nginx: worker"

Locate the process ID of the worker you want to probe. In my case there is one worker with process ID 20192. To start GDB and load the worker process, run this command (note that the worker process is paused while GDB is running):

# sudo gdb --pid 20192

To load the script you downloaded from GitHub Gist and dump the debug log, run these commands:

(gdb) source nginx.gdb
(gdb) ddl

GDB creates a file called debug_log.txt containing a dump of the memory allocated using the error_log directive, so in our case the file is 32 MB in size. If there are only a few entries in the file, you can easily to truncate it with the following sed command. In most cases you don’t need to do this, and if the log has already wrapped back to the beginning, the command has no effect:

# sed -i 's/[[:space:]]*$//' debug_log.txt

Obviously it is not always a good idea to have the worker process paused while grabbing the log. So you can tell GDB to grab the log and exit right away, limiting the pause to a very short time. This approach is very similar to the excellent Poor Man’s Profiler from an old colleague of mine, Domas Mituzas.

# gdb --pid 20192 -iex "source nginx.gdb" -ex "ddl" --batch

Dumping the Active NGINX Configuration

In NGINX 1.9.2 and later, the entire configuration is stored in memory when NGINX has been built using the --with-debug configuration option, making it possible to extract the configuration from the master process with GDB. This can be useful, both to verify which configuration has been loaded and to restore a previous configuration if the version on disk has been accidentally removed or overwritten.

As before, the nginx.gdb file from the GitHub Gist has the function you need to run the memory dump. So, first we load GDB:

# sudo gdb --pid `pgrep -f "nginx: master"`

Then we run the following commands to dump the configuration.

(gdb) source nginx.gdb
(gdb) dcfg

GDB spits out the names of the files as it dumps them. As shown in this sample output, there might be a bit of junk at the end of each filename, because GDB’s printf function doesn’t work all that well when strings are not NUL terminated. The final result is a file called nginx_conf.txt containing the complete active configuration.

Using GDB with Core Files

Everything we have covered in this post can also be used with core files to aid with debugging the cause of issues:

# gdb --core core.9491 nginx

Both the dcfg and ddl functions described in this blog post can be used with core files in the way we’ve discussed here. This can be useful if you need to find the configuration of the NGINX server at the time the core file was generated or to get debugging information for the events leading up to the generation of the core file.

Summary

Dumping the debug log and the configuration can both be really useful ways to extract information about NGINX internals, and of course you can expand on the techniques I’ve introduced by tweaking the GDB script. For example, when dumping the configuration it should be possible to dump each loaded configuration file into a separate output file rather than dumping everything into a single file. The length of the filenames is stored with the filenames themselves, so there should be a way to either copy them or truncate them when using them. I haven’t figured out a good way of doing this with the standard scripting API, but recent versions of GDB support Python scripting which gives you the ability to do this.

It’s worth repeating that we recommend you use these techniques only in development and testing environments. It is a not a great idea to pause the execution of NGINX processes, especially the workers, in a production environment.

I’d be most interested in any GDB recipes you come up with – please share them in the comments.