Top Link Bar

Quick Launch

Rate It:

Average Rating:

(4 Ratings)

Cause and Analysis of Unix Core files

Content

INFA_Content

Fatal run-time errors in UNIX programs usually result in thetermination of the UNIX process by the operating system. Usually,when the operating system terminates a process, a 'core dump' file is also created, which can be used to analyze the reason for the abnormal termination.

What is a 'Core' File and What Causes it to be Created?

UNIX operating systems may terminate a process before it's normal, expected exit for several reasons. These reasons are typically for bad behavior by the program, and include attempts to execute illegal or incorrect machine instructions, attempts to allocate memory outside the memory space allocated to the program, attempts to write to memory marked read-only by the operating system and other similar incorrect low level operations. Most of these bad behaviors are caused by errors in programming logic in the program.

UNIX may also terminate a process for some reasons that are not caused by programming errors. The main examples of this type of termination are when a process exceeds it's CPU time limit, and when a process exceeds it's memory limit.

When UNIX terminates a process in this way, it normally writes an image of the processes memory to disk in a single file. These files are called 'core files', and are intended to be used by a programmer to help determine the cause of the failure. Depending on the UNIX version, the name of the file will be 'core', or in more recent UNIX versions, it is 'core.nnnn' where nnnn is the UNIX process ID of the process that was terminated.

Core files are not created for 'normal' runtime errors such as incorrect file permissions, lack of disk space, inability to open a file or network connection, and other errors that a program is expected to detect and handle. However, under certain error conditions a program may not handle the error conditions correctly and may follow a path of execution that causes the OS to terminate it and cause a core dump.

Mixing incompatible versions of UNIX, vendor, and database libraries can often trigger behavior that causes unexpected core dumps. For example, using an ODBC driver library from one vendor and an ODBC driver manager from another vendor may result in a coredump if the libraries are not compatible. A similar situation can occur if a process is using libraries from different versions of a database client, such as a mixed installation of Oracle 8i and 9i. An installation like this should not exist, but if it does, core dumps are often the result.

Core File Locations and Size Limits

A core file is written to the current working directory of the process that was terminated. For PowerCenter, this is always the directory the server was started from. For other applications, this may not be true.

UNIX also implements a per user resource limit on the maximum size of core files. This is controlled by the ulimit command. If the limit is 0, then core files will not be created. If the limit is less than the total memory size of the process, a partial core file will be written. Refer the Best Practice on UNIX resource limits.

Analyzing Core Files

There is little information in a core file that is relevant to an end user; most of the contents of a core file are only relevant to a developer, or someone who understands the internals of the program that generated the core file. However, there are a few things that an end user can do with a core file in the way of initial analysis.

The first step is to use the UNIX 'file' command on the core, which will show which program generated the core file:

Core files can be generated by both the PowerCenter executables(i.e., pmserver, pmrepserver, and pmdtm) as well as from other UNIX commands executed by the server, typically from command tasks and per- or post-session commands. If a PowerCenter process is terminated by the OS and a core is generated, the session or server log typically indicates 'Process terminating on Signal/Exception' as it's last entry.

Using the pmstack Utility

Informatica provides a 'pmstack' utility, which canautomatically analyze a core file. If the core file is fromPowerCenter, it will generate a complete stack trace from the core file, which can be sent to Informatica Customer support for furtheranalysis. The track contains everything necessary to furtherdiagnose the problem. Core files themselves are normally not useful on a system other than the one where they were generated.

The pmstack utility can be downloaded from the InformaticaSupport knowledge base as article 13652, and from the support FTP server at tsftp.informatica.com. Once downloaded, run pmstack with the -c option, followed by the name of the core file:

Pmstack also supports a -p option, which can be used to extract a stack trace from a running process. This is sometimes useful if the process appears to be hung, to determine what the process is doing.