HowTo: Debug Crashed Linux Application Core Files Like A Pro

byVivek GiteonJune 3, 2010

Core dumps are often used to diagnose or debug errors in Linux or UNIX programs. Core dumps can serve as useful debugging aids for sys admins to find out why Application like Lighttpd, Apache, PHP-CGI or any other program crashed. Many vendors and open source project author requests a core file to troubleshoot a program. A core file is generated when an application program abnormally terminates due to bug, operating system security protection schema, or program simply try to write beyond the area of memory it has allocated, and so on. This article explains how to turn on core file support and track down bugs in programs.

Turn On Core File Creation Support

By default most Linux distributions turn off core file creation (at least this is true for RHEL, CentOS, Fedora and Suse Linux). You need to use the ulimit command to configure core files.

See The Current Core File Limits

Type the following command:# ulimit -c Sample outputs:

0

The output 0 (zero) means core file is not created.

Change Core File Limits

In this example, set the size limit of core files to 75000 bytes:# ulimit -c 75000

fs.suid_dumpable = 2 - Make sure you get core dumps for setuid programs.

kernel.core_pattern = /tmp/core-%e-%s-%u-%g-%p-%t - When the application terminates abnormally, a core file should appear in the /tmp. The kernel.core_pattern sysctl controls exact location of core file. You can define the core file name with the following template whih can contain % specifiers which are substituted by the following values when a core file is created:

How Do I Read Core Files?

You need use the gdb command as follows:$ gdb /path/to/application /path/to/corefile See the gdb command man page for more information.

strace command

System administrators, diagnosticians and trouble-shooters will find it invaluable for solving problems with programs for which the source is not readily available since they do not need to be recompiled in order to trace them. This is also useful to submit bug reports to open source developers. See how to use the strace command under Linux to debug the problems.

I’m returning to *nix programming after a long absence, and was trying to understand the use of DAEMON_COREFILE_LIMIT. What I haven’t seen mentioned is that this environment variable is used by the shell script function ‘daemon()’ which is included in the group of functions in etc/init.d/functions. If you start your daemon by calling daemon(), that variable will get used in a call to ‘ulimit’ by the shell. Otherwise, it will have no effect. I haven’t verified this yet (getting access to our servers is a slow process), but it seems that our custom shell script to start the daemon can do the same thing by calling ulimit directly.