Linux Applications Debugging Techniques/The debugger

Someday some hard to reproduce issue will be found on a production machine. Typically, such a machine is difficult to access, has no development environment and nothing can be installed on it. At most it will have gdb, but very likely not. Typically also, the heaviest user will be the one to find the issue and typically still, the heaviest user is the one being the bigger -- money-wise. And that issue has to be root cause diagnosed and fixed.

Unless the application has been prepared for this forensic gathering moment, not much can be done to diagnose where the issue is. Thus, preparations should start with compilation:

Have a "symbol server" and make sure it is reacheable. Compile on the symbol server.

Compile the release with debugging symbols. Strip them if you do not want to ship them but keep them.

Ship gdbserver with the application for remote debugging.

These preparations will allow one to:

Debug the application running on any machine, including machines where there is no gdb installed.

Debug from any machine that has network visibility to the symbol server.

Also, think beforehand how would you debug on the machine:

Embed a breakpoint in the code, at hard of reach places of interest, then

One way to easily reach the right code from within the debugger is to build the binaries within an auto-mounted folder, each build in its own sub-folder. The same auto-mount share should be accessible from the machine you are debugging on.

Watchpoints can be implemented either in software either in hardware if the CPU supports it. Typically on an Intel processor there are eight debug registers out of which only four can be used for hardware breakpoints and this limits the number of watchpoints system wide.

As a note, in upcoming gdb releases, .gdbinit will be replaced by gdb-gdb.gdb:

gdb-gdb.gdb
^ ^ ^
^ ^ ^---- It's a gdb script.
^ ^ If it were Python this would be .py.
^ ^
^ ^-------- "-gdb" is a gdb convention, it's the suffix added to a file
^ for auxiliary support.
^ E.g., gdb will auto-load libstdc++.so-gdb.py (version elided)
^ which contains the std c++ pretty-printers.
^
^------------ This init script is for the program named "gdb".
If this were for readelf the script would be named
readelf-gdb.gdb.