Revision as of 17:11, 9 May 2012

Ftrace is the Linux kernel internal tracer that was included in the Linux kernel in 2.6.27.
Although Ftrace is named after the function tracer it also includes many more functionalities.
But the function tracer is the part of Ftrace that makes it unique as you can trace almost
any function in the kernel and with dynamic Ftrace, it has no overhead when not enabled.

The interface for Ftrace resides in the debugfs file system in the tracing directory.
Documentation for this can be found in the Linux kernel source tree at
Documentation/trace/ftrace.txt.

Also within that same repository is KernelShark, which is a graphical user interface to trace-cmd. trace-cmd
is built with just "make" and KernelShark is created with "make gui". This allows building trace-cmd on
your embedded device and keeping the build from needing the GTK libraries required by KernelShark.

Tips

Tracing a specific process with the Ftrace interface

(Adapted from email by Steven Rostedt)
To trace just the kernel functions executed in the context of a particular function,
set the pseudo-variable 'set-ftrace-pid', to the process id (pid) of the process.

If the process is not already running, you can use a wrapper shell script and the
'exec' command, to execute a command as a known pid.

In this example, '$$' is the pid of the currently executing process (the shell
script. This is set into the 'set_ftrace_pid' variable, then the 'function'
tracer is enabled. Then this script exec's the command (specified by the first
argument to the script).

Example usage (assuming script is called 'trace_command'):

trace_command ls

Tracing a specific process with trace-cmd

# trace-cmd record -p function -F ls

Capturing an oops (from startup) to the serial console

You can capture the function calls leading up to a panic
by placing the following on the kernel command line:

ftrace=function ftrace_dump_on_oops

Note: You can also use 'ftrace=function_graph' if you would prefer that instead.

The ftrace documentation, in Documentation/trace/ftrace.txt mentions how to set
ftrace_dump_on_oops in a running system, but I have found it very handy to have
it configured to dump the trace from kernel startup, so that any panics that occur
during boot (before user-space is started) are also captured.

Find latencies on kernel startup

It is possible to use ftrace to record functions that exceed a certain amount of time, using
the 'tracing_thresh' option.
This can be used for finding routines that are taking a long time
on kernel startup, to help optimize bootup time:

Make sure the following kerne configuration options are set:

CONFIG_FTRACE: "Tracers"

CONFIG_FUNCTION_TRACER: "Kernel Function Tracer"

CONFIG_FUNCTION_GRAPH_TRACER: "Kernel Function Graph Tracer"

Use the following on the kernel command line:

tracing_thresh=200 ftrace=function_graph

this traces all functions taking longer than 200 microseconds (.2 ms). You can use any duration threshhold you want.

to get the data:

$ mount -t debugfs debugfs /debug

$ cat /debug/tracing/trace

These command should be probably be done programatically (as part of an init script), to avoid data loss

scope of operation

the tracer starts sometime during initialization, and you only get timings after it starts

find deepest kernel stack

This is useful to find the routine with the deepest kernel stack

set the following kernel configuration options:

CONFIG_FTRACE: "Tracers"

CONFIG_FUNCTION_TRACER: "Kernel Function Tracer"

CONFIG_STACK_TRACER: "Trace max stack"

use the following on the kernel command line:

stacktrace

to get the data:

$ mount -t debugfs debugfs /debug

$ cat /debug/tracing/stack_trace

scope of operation

the stack tracer will continue operating until you turn it off, which can be done with: