Synopsis

Description

The prex command is the part of the Solaris tracing architecture that
controls probes in a process or the kernel. See tracing(3TNF) for an overview
of this tracing architecture, including example source code using it.

prex is the application used for external control of probes. It automatically
preloads the libtnfprobe library. prex locates all the probes in a target
executable or the kernel and provides an interface for the user to
manipulate them. It allows a probe to be turned on for tracing, debugging,
or both. Tracing generates a TNF (Trace Normal Form) trace file that
can be converted to ASCII by tnfdump(1) and used for performance analysis.
Debugging generates a line to standard error whenever the probe is hit
at run time.

prex does not work on static executables. It only works on dynamic
executables.

Invoking prex

There are three ways to invoke prex:

Use prex to start the target application cmd. In this case, the target application need not be built with a dependency on libtnfprobe. See TNF_PROBE(3TNF). prex sets the environment variable LD_PRELOAD to load libtnfprobe into the target process. See ld(1). prex then uses the environment variable PATH to find the target application.

Attach prex to a running application. In this case, the running target application should have libtnfprobe already linked in. Alternatively, the user may manually set LD_PRELOAD to include libtnfprobe.so.1 prior to invoking the target.

Use prex with the -k option to set prex to kernel mode. prex can then be used to control probes in the Solaris kernel. In kernel mode, additional commands are defined, and some commands that are valid in other modes are invalid. See KernelMode below.

Control File Format and Command Language

In a future release of prex, the command language may be moved
to a syntax that is supported by an existing scripting language like
ksh(1). In the meantime, the interface to prex is uncommitted.

Commands should be in ASCII.

Each command is terminated with the NEWLINE character.

A command can be continued onto the next line by ending the previous line with the backslash (“\”) character.

Tokens in a command must be separated by whitespace (one or more spaces or tabs).

The "#" character implies that the rest of the line is a comment.

Basic prex Commands

Command

Result

% prex a.out

Attaches prex to your program and starts prex.

prex> enable $all

Enables all the
probes.

prex> quit resume

Quits prex and resumes execution of program.

Control File Search Path

There are two different methods of communicating with prex:

By specifications in a control file. During start-up, prex searches for a file named .prexrc in the directories specified below. prex does not stop at the first one it finds. This way a user can override any defaults that are set up. The search order is:

$HOME/
./

By typing commands at the prex prompt.

The command language for both methods is the same and is specified
in USAGE. The commands that return output will not make sense in
a control file. The output will go to standard output.

When using prex on a target process, the target will be in
one of two states, running or stopped. This can be detected by
the presence or absence of the prex> prompt. If the prompt is
absent, it means that the target process is running. Typing Control-C will stop
the target pr ocess and return the user to the prompt. There
is no guarantee that Control-C will return to a prex prompt immediately.
For example, if the target process is stopped on a job control
stop (SIGSTOP), then Control-C in prex will wait until the target has been
continued (SIGCONT). See Signals to Target Program below for more information on signals and the
target process.

Options

The following options are supported:

-k

kernel mode: prex is used to control probes in the Solaris kernel. In kernel mode, additional commands are defined, and some commands valid in other modes are invalid. See Kernel Mode below.

-llibraries

The libraries mentioned are linked in to the target application using LD_PRELOAD (see ld(1)). This option cannot be used when attaching to a running process. The argument to the -l option should be a space-separated string enclosed in double quotes. Each token in the string is a library name. It follows the LD_PRELOAD rules on how libraries should be specified and where they will be found.

-otrace_file_name

File to be used for the trace output. trace_file_name is assumed to be relative to the current working directory of prex (that is, the directory that the user was in when prex was started).

If prex attaches to a process that is already tracing, the new trace_file_name (if provided) will not be used. If no trace_file_name is specified, the default is /$TMPDIR/trace-pid where pid is the process id of the target program. If TMPDIR is not set, /tmp is used.

-skbytes_size

Maximum size of the output trace file in Kbytes. The default size of the trace kbytes_size is 4096 (210) bytes or 4 Mbytes for normal usage, and 384 or 384 kbytes in kernel mode. The minimum size that can be specified is 128 Kbytes. The trace file can be thought of as a least recently used circular buffer. Once the file has been filled, newer events will overwrite the older ones.

Usage

This section describes the usage of the prex utility.

Grammar

Probes are specified by a list of space-separated selectors. Selectors are of
the form:

attribute=value

(See TNF_PROBE(3TNF)). The “attribute=” is optional. If it is not specified, it
defaults to “keys=”.

The attribute or value (generically called “spec”) can be any of the
following:

The connect command is used to connect probe functions (which must be prefixed by `&') to probes. The probes are specified either as a single set (with a `$'), or by explicitly listing the probe selectors in the command. The probe function has to be one that is listed by the list fcns command. This command does not enable the probes.

The clear command is used to disconnect all connected probe functions from the specified probes.

The trace and untrace commands are used to toggle the tracing action of a probe point (that is, whether a probe will emit a trace record or not if it is hit). This command does not enable the probes specified. Probes have tracing on by default. The most efficient way to turn off tracing is by using the disable command. untrace is useful if you want debug output but no tracing. If so, set the state of the probe to enabled, untraced, and the debug function connected.

The enable and disable commands are used to control whether the probes perform the action that they have been set up for. To trace a probe, it has to be both enabled and traced (using the trace command). Probes are disabled by default. The list history command is used to list the probe control commands issued: connect, clear, trace, untrace, enable, and disable. These are the commands that are executed whenever a new shared object is brought in to the target program by dlopen(3C). See the subsection, dlopen'ed Libraries, below for more information.

The following table shows the actions that result from specific combinations of tracing, enabling, and connecting:

Enabled or Tracing State Debug State Results
Disabled (On/Off) (Connected/Cleared) In
------------------------------------------------------------
Enabled On Connected Tracing and
Debugging
Enabled On Cleared Tracing only
Enabled Off Connected Debugging only
Enabled Off Cleared Nothing
Disabled On Connected Nothing
Disabled On Cleared Nothing
Disabled Off Connected Nothing
Disabled Off Cleared Nothing

List History

list history # lists probe control command history

The list history command displays a list of the probe control commands previously issued in the tracing session, for example, connect, clear, trace, disable. Commands in the history list are executed wherever a new shared object is brought into the target program by dlopen(3C).

The first two commands list the selected attributes and values of the specified probes. They can be used to check the state of a probe. The third command lists the various values associated with the selected attributes. The fourth command lists the current tracefile.

Help Command

help topic

To get a list of the help topics that are available, invoke the help command with no arguments. If a topic argument is specified, help is printed for that topic.

Source a File

source filename

The source command can be used to source a file of prex commands. source can be nested (that is, a file can source another file). filename is a quoted string.

The default quit will continue the target process if prex attached to it. Instead, if prex had started the target program, quit will kill the target process.

dlopen'ed Libraries

Probes in shared objects that are brought in by dlopen(3C) are automatically
set up according to the command history of prex. When a shared object
is removed by a dlclose(3C), prex again needs to refresh its understanding
of the probes in the target program. This implies that there is more
work to do for dlopen(3C) and dlclose(3C) —so they will take slightly
longer. If a user is not interested in this feature and doesn't
want to interfere with dlopen(3C) and dlclose(3C), detach prex from the target
to inhibit this feature.

Signals to Target Program

prex does not interfere with signals that are delivered directly to the
target program. However, prex receives all signals normally generated from the terminal,
for example, Control-C (SIGINT), and Control-Z (SIGSTOP), and does not forward them to
the target program. To signal the target program, use the kill(1) command
from a shell.

Interactions with Other Applications

Process managing applications like dbx, truss(1), and prex cannot operate on the
same target program simultaneously. prex will not be able to attach to
a target which is being controlled by another application. A user can
trace and debug a program serially by the following method: first attach
prex to target (or start target through prex), set up the probes
using the command language, and then type quit suspend. The user can then attach
dbx to the suspended process and debug it. A user can also
suspend the target by sending it a SIGSTOP signal, and then by
typing quit resume to prex. In this case, the user should also send
a SIGCONT signal after invoking dbx on the stopped process (else dbx
will be hung).

Failure of Event Writing Operations

There are a few failure points that are possible when writing out
events to a trace file, for example, system call failures. These failures
result in a failure code being set in the target process. The
target process continues normally, but no trace records are written. Whenever a
user enters Control-C to prex to get to a prex prompt, prex will
check the failure code in the target and inform the user if
there was a tracing failure.

Target Executing a Fork or exec

If the target program does a fork(2), any probes that the child
encounters will cause events to be logged to the same trace file.
Events are annotated with a process id, so it will be possible to
determine which process a particular event came from. In multi-threaded programs, there
is a race condition with a thread doing a fork while the
other threads are still running. For the trace file not to
get corrupted, the user should either use fork1(2), or make sure that
all other threads are quiescent when doing a fork(2),

If the target program itself (not any children it may fork(2)) does
an exec(2), prex detaches from the target and exits. The user can
reconnect prex with prex-ppid.

A vfork(2) is generally followed quickly by an exec(2) in the
child, and in the interim, the child borrows the parent's process while
the parent waits for the exec(2). Any events logged by the child from
the parent process will appear to have been logged by the parent.

Kernel Mode

Invoking prex with the -k flag causes prex to run in kernel mode.
In kernel mode, prex controls probes in the Solaris kernel. See tnf_kernel_probes(4)
for a list of available probes in the Solaris kernel. A few
prex commands are unavailable in kernel mode; many other commands are valid
in kernel mode only.

The -l, -o, and -p command-line options are not valid in kernel
mode (that is, they may not be combined with the -k flag).

The rest of this section describes the differences in the prex command
language when running prex in kernel mode.

prex will not stop the kernel

When prex attaches to a running user program, it stops the user program. Obviously, it cannot do this when attaching to the kernel. Instead, prex provides a ``tracing master switch'': no probes will have any effect unless the tracing master switch is on. This allows the user to iteratively select probes to enable, then enable them all at once by turning on the master switch.

The command

ktrace [ on | off ]

is used to inspect and set the value of the master switch. Without an argument, prex reports the current state of the master switch.

Since prex will not stop or kill the kernel, the

quit resume

and

quit kill

commands are not valid in kernel mode.

No functions may be attached to probes in the kernel

In particular, the debug function is unavailable in kernel mode.

Trace output is written to an in-core buffer

In kernel mode, a trace output file is not generated directly, in order to allow probes to be placed in time-critical code. Instead, trace output is written to an in-core buffer, and copied out by a separate program, tnfxtract(1).

The in-core buffer is not automatically created. The following prex command controls buffer allocation and deallocation:

buffer [ alloc [ size ] | dealloc ]

Without an argument, the buffer command reports the size of the currently allocated buffer, if any. With an argument of alloc [size], prex allocates a buffer of the given size. size is in bytes, with an optional suffix of 'k' or 'm' specifying a multiplier of 1024 or 1048576, respectively. If no size is specified, the size specified on the command line with the -s option is used as a default. If the -s command line option was not used, the ``default default'' is 384 kilobytes.

With an argument of dealloc, prex deallocates the trace buffer in the kernel.

prex will reject attempts to turn the tracing master switch on when no buffer is allocated, and to deallocate the buffer when the tracing master switch is on. prex will refuse to allocate a buffer when one is already allocated; use buffer dealloc first.

prex will not allocate a buffer larger than one-half of a machine's physical memory.

prex supports per-process probe enabling in the kernel

In kernel mode, it is possible to select a set of processes for which probes are enabled. No trace output will be written when other processes traverse these probe points. This is called "process filter mode". By default, process filter mode is off, and all processes cause the generation of trace records when they hit an enabled probe.

Some kernel events such as interrupts cannot be associated with a particular user process. By convention, these events are considered to be generated by process id 0.

prex provides commands to turn process filter mode on and off, to get the current status of the process filter mode switch, to add and delete processes (by process id) from the process filter set, and to list the current process filter set.

The process filter set is maintained even when process filter mode is off, but has no effect unless process filter mode is on.

When a process in the process filter set exits, its process id is automatically deleted from the process filter set.

The command:

pfilter [ on | off | add pidlist | delete pidlist ]

controls the process filter switch, and process filter set membership. With no arguments, pfilter prints the current process filter set and the state of the process filter mode switch:

on or off

set the state of the process filter mode switch.

addpidlist

deletepidlist

add or delete processes from the process filter set. pidlist is a comma-separated list of one or more process ids.

Examples

See tracing(3TNF) for complete examples showing, among other things, the use of
prex to do simple probe control.

When either the process or kernel is started, all probes are disabled.

Notes

Currently, the only probe function that is available is the &debug function.
When this function is executed, it prints out the arguments sent in
to the probe as well as the value associated with the sunw%debug
attribute in the detail field (if any) to stderr.

Some non-SPARC hardware lacks a true high-resolution timer, causing gethrtime() to return
the same value multiple times in succession. This can lead to problems
in how some tools interpret the trace file. This situation can be
improved by interposing a version of gethrtime(), which causes these successive values
to be artificially incremented by one nanosecond:

Of course, this does not increase the resolution of the timer, so
timestamps for individual events are still relatively inaccurate. But this technique maintains
ordering, so that if event A causes event B, B never appears
to happen before or at the same time as A.

dbx is available with the Sun Workshop Products.

Bugs

prex should issue a notification when a process id has been automatically
deleted from the filter set.

There is a known bug in prex which can result in this
message:

Tracing shut down in target program due to an internal
error - Please restart prex and target

When prex runs as root, and the target process is not root,
and the tracefile is placed in a directory where it cannot be
removed and re-created (a directory with the sticky bit on, like /tmp),mm
then the target process will not be able to open the tracefile when
it needs to. This results in tracing being disabled.

Changing any of the circumstances listed above should fix the problem. Either
don't run prex as root, or run the target process as root,
or specify the tracefile in a directory other than /tmp.