This technical note describes how to use SunTM Gathering Debug Data (Sun GDD or GDD) to collect data that the Sun Support Center requires in order to debug problems with Sun JavaTM System Directory Server software.

By collecting this data before you open a Service Request, you can reduce
substantially the time needed to analyze and resolve the problem. For more
information on how this document and associated scripts can help you in better
dealing with Directory Server problems, see:

1.1 Technical Note Revision History

1.2 About This Technical Note

This document covers Sun Java System Directory Server 5 on all supported
platforms.

You can use this document in all types of environments, including test,
pre-production, and production. Verbose debugging is not used so as to avoid
performance impact, except when it is deemed necessary. In some cases, it
is possible that the problem could disappear when you configure logging for
debug mode. However, this is the minimum to understand the problem. In the
majority of cases, the debug data described in this document is sufficient
to analyze the problem.

This document does not provide workarounds nor techniques or tools to
analyze debug data. It provides some troubleshooting, but you should not use
this guide as an approach to troubleshooting Directory Server problems.

See To Run the pkg_app Script for
instructions on using pkg_app. See the documentation bundled
with dirtracer for information on using that script.

On the Windows platform, download the free Debugging Tools
for Windows to help in analyzing process hang problems. The debugger Dr. Watson
is not useful for process hang problems because it cannot generate a crash
dump on a running process. Download the free Debugging Tools from the following
location.

When opening a Service Request by phone with the Sun Support Center,
provide a summary of the problem. Also provide the details in a text file
named Description.txt. Be sure to include Description.txt in the archive along with the rest of your debug data.

If these log files are not in the default locations,
examine the Directory Server configuration file, server-root/slapd-/serverID/config/dse.ldif,
to find the paths to the logs. The paths are specified as the values of attributes nsslapd-accesslog, nsslapd-errorlog, and nsslapd-auditlog.

This section describes what data to collect when you cannot complete Directory Server installation.

Before You Begin

If the problem concerns a general installation failure for Java Enterprise System,
first check the installation troubleshooting chapter in the Installation
Guide for your version of Java Enterprise System.

The log file is named after the date and time that the installation
failed. For example, a log file for an installation that failed on December
16 at 3:32 p.m. would have a name like Java_Enterprise_System*_install.B12161532.

On Solaris systems, installation logs are located
under /var/sadm/install/logs.

On Red Hat
and HP-UX systems, installation logs are located under /var/opt/sun/install/logs.

On Windows systems, installation logs are located
under C:\Documents and Settings\current-user\Local
Settings\Temp.

To Collect Required Debug Data For
an Unresponsive or Hung Directory Server Process

This procedure describes what data to collect when Directory Server is
still running, but is no longer responding to client application requests.

Collect the data describe in this procedure while the server
is hanging.

Note the time during which the hang is seen to occur.

Collect information about the port used during the hang.

UNIX and Linux

netstat -an | grep ds-port

Windows

netstat -an

Collect statistics about the system running Directory Server.

Solaris OS

ps -aux | grep server-root

vmstat 5 5

iostat -x

top

uptime

HP-UX

ps -aux | grep server-root

vmstat 5 5

iostat -x

top

sar

Red Hat

ps -aux | grep server-root

vmstat 5 5

top

uptime

sar

Windows

Get the process ID using the tlist.exe command,
then get process details using the same command.

win-dbg-root\tlist.exe pid

Collect swap information.

Solaris OS

swap -l

HP-UX

swapinfo

Red Hat

free

Windows

Already provided in C:\report.txt.

On Solaris systems, collect output from pstack and pmap five times, once every ten seconds.

pstack pid

pmap -x pid

Get output showing system calls during the hang, by letting each
of the commands listed here run for about a minute, then stopping them by
typing Control-C.

When the server is hanging, attempt to get several core files
that show the state of the server threads over time. You can do this by generating
a core file, changing the name of the core file, waiting
30 seconds to a minute, and generating another core file.
Repeat the process at least once to get a minimum of three sets of core files
and related data.

Solaris OS

cd server-root/bin/slapd/server

gcore -o /tmp/directory-core pid

pstack /tmp/directory-core

For each core file on Solaris OS, collect output from the following
commands.

cd server-root/bin/slapd/server

file core-file

pstack core-file

pmap core-file

pflags core-file

For at least one of the core files on Solaris OS, collect output from
the pkg_app script.

If you cannot find the change log, examine the Directory Server configuration
file, server-root/slapd-serverID/config/dse.ldif, to find the path. The path is specified
as the values of attribute nsslapd-changelogdir.

Collect output from the insync command.

The insync command indicates the state of synchronization
between a master replica and one or more consumer replicas. The following
command shows the state over a period of 30 seconds.

To Collect Required Debug Data
For Directory Server Schema Problems

Collect a copy of the schema folder, server-root/slapd-serverID/config/schema,
and all the files in the folder.

Indicate whether you have the same problem both in the Console
and on the command line.

Provide a list of the last changes made to the schema files.

If the schema violation occurred during an add, modify, delete,
or replace operation, provide an LDIF representation of the changes and a
list of the commands used.

1.6 Configuring the Operating System to Generate
Core Files

Core files and crash dumps are generated when a process or application
terminates abnormally. You can also generate core files and crash dumps to
help diagnose why a process does not respond to client application requests.

To Configure Solaris OS to Generate Core
Files

This procedure shows you how to use the coreadm command
to configure the system so that all process core files are placed in a single
system directory. This means it is easier to track problems by examining core
files in a specific directory whenever a Solaris OS process or daemon terminates
abnormally.

Before You Begin

Make sure that the /var file system where the core
files are generated has sufficient space. Once you configure Solaris OS to
generate core files as shown here, all processes that crash write a core file
to the /var/cores directory.

Specifies the global core file name pattern. Unless a per-process
pattern or setting overrides it, core files are stored in the specified directory
with a name such as program.node.pid.time.core, for example: mytest.myhost.1234.1102010309.core.

Capability of setuid programs to also dump
core as per the same pattern

Generation of a syslog message by any attempt to dump core
(successful or not)

-d

Specifies options to disable. The preceding command disables:

Core dumps per the per-process core file pattern

Per-process core dumps of setuid programs

The preceding command stores all core dumps in a central location with
names identifying what process dumped core and when. These changes only impact
processes started after you run the coreadm command. Use
the coreadm -u command after the preceding command to apply
the settings to all existing processes.

When
providing crash dumps, collect both the dmp and drwtsn32.log files.

Use the Window Debugging Tools to generate crash dumps of a running
process.

Make sure you install the latest version of the Debugging Tools
and OS Symbols for your version of Windows.

Set the _NT_SYMBOL_PATH for your environment.

Enable generation of a crash dump for your application.

Get
the process ID of the application using the tlist.exe command,
then enable the crash dump.

win-dbg-root\tlist.exe

win-dbg-root\adplus.vbs
-crash -FullOnFirst -p pid -o C:\dump-dir

The adplus.vbs command
tracks the application with process ID pid. The adplus.vbs command generates a dmp file in
the event of a crash.

When collecting crash dump information, take the complete folder
generated under C:\dump-dir.

To Run the pkg_app Script

The pkg_app script packages an executable and all
of its shared libraries into one compressed tar file given the process ID
of the application, and optionally the name of the core file to be opened.
The files are stripped of their directory paths, and are stored under a relative
directory named app/ with their names only, allowing
them to be unpacked in one directory.

On Solaris 9 and Solaris 10, the list of files is derived from the core
file rather than the process image if it is specified. You must still provide
the process ID of the running application to assist in path resolution.

Two scripts are created to facilitate opening the core file when the
tar file is unpacked.

opencore. This is the script to be executed
once unpacked. It sets the name of the core file and the linker path to use
the app/ directory, and then invokes dbx with
the dbxrc file as the argument.

dbxrc. This script contains the dbx initialization
commands to open the core file.

Copy the script to a temporary directory on the system where the
server is installed.

Become superuser.

Run the pkg_app script in one of the following
ways.

./pkg_app server-pidcore-file

./pkg_app server-pid

The pkg_app script prompts for the core-file name.

./pkg_app core-file

1.7 Reporting Problems

Use the following email aliases to report problems with this document
and its associated scripts:

1.9 Third-Party Web Site References

Third-party URLs are referenced in this document and provide additional,
related information.

Note –

Sun is not responsible for the availability of third-party web
sites mentioned in this document. Sun does not endorse and is not responsible
or liable for any content, advertising, products, or other materials that
are available on or through such sites or resources. Sun will not be responsible
or liable for any actual or alleged damage or loss caused or alleged to be
caused by or in connection with use of or reliance on any such content, goods,
or services that are available on or through such sites or resources.

1.10 Searching Sun Product
Documentation

Besides searching for Sun product documentation from the docs.sun.com
web site, you can use a search engine of your choice by typing the following
syntax in the search field:

search-term site:docs.sun.com

For example, to search for Directory Server, type
the following:

"Directory Server" site:docs.sun.com

To include other Sun web sites in your search, such as java.sun.com,
www.sun.com, and developers.sun.com, use sun.com in place
of docs.sun.com in the search field.

1.11 Sun Welcomes Your Comments

Sun is interested in improving its documentation and welcomes your comments
and suggestions. To share your comments, go to http://docs.sun.com and click Send Comments. In the online form, provide the
full document title and part number. The part number is a 7-digit or 9-digit
number that can be found on the book's title page or in the document's URL.
For example, the part number of this book is 820-0437-10.