Starting from 2.6.28 the kernel has this new feature to optimize the boot time. It records the timings of the initcalls. Its aim is to be parsed by the scripts/bootgraph.pl tool to produce graphics about boot inefficiencies, giving a visual representation of the delays during initcalls. Users need to boot with the "initcall_debug" and "printk.time=1" parameters, and run "dmesg | perl scripts/bootgraph.pl > output.svg" to generate the final data.

=== Kernel Function Instrumentation (KFI) ===

=== Kernel Function Instrumentation (KFI) ===

Line 47:

Line 51:

=== ftrace ===

=== ftrace ===

−

Ftrace is a simple function tracer which initially came from the -rt patches but was mainlined in 2.6.27. Compiler profiling features are used to insert an instrumentation call that can be overwritten with a NOP sequence to ensure overhead is minimal with tracing disabled. There are a number of tracers in the kernel that use ftrace to trace high level events such as irq enabling/disabling preemption enabling/disabling, scheduler events and branch profiling.

+

Ftrace is a simple function tracer which initially came from the -rt patches but was mainlined in 2.6.27. Compiler profiling features are used to insert an instrumentation call (with gcc ''-pg'' option) that can be overwritten with a NOP sequence to ensure overhead is minimal with tracing disabled (this is enabled through CONFIG_DYNAMIC_FTRACE). There are a number of tracers in the kernel that use ftrace to trace high level events such as irq enabling/disabling preemption enabling/disabling, scheduler events and branch profiling.

−

The interface to access ftrace can be found in /debugfs/tracing, and is documented in Documentation/ftrace.txt.

+

The interface to access ftrace can be found in ''/debugfs/tracing'', and is very extensively documented in [http://lxr.free-electrons.com/source/Documentation/trace/ftrace.txt Documentation/trace/ftrace.txt].

+

+

Steven Rostedt, ftrace's main developer, gave a conference on ftrace at the Ottawa Linux Symposium 2008. The [http://free-electrons.com/pub/video/2008/ols/ols2008-steven-rostedt-ftrace.ogg video] and [http://people.redhat.com/srostedt/ftrace-tutorial.odp slides].

=== SystemTap / Kprobes ===

=== SystemTap / Kprobes ===

−

[http://sourceware.org/systemtap/ SystemTap] is a sophisticated kernel instrumentation tool that can be scripted with it's own language to gather information about a running kernel. It uses the Kprobes infrastructure to implement it's tracing.

+

[http://sourceware.org/systemtap/ SystemTap] is a sophisticated kernel instrumentation tool that can be scripted with its own language to gather information about a running kernel. It uses the Kprobes infrastructure to implement it's tracing.

== Notes ==

== Notes ==

Line 63:

Line 69:

**instrumentation systems tend to pollute the cache lines for the processor

**instrumentation systems tend to pollute the cache lines for the processor

*There doesn't seem to be a single API to support in-kernel timing instrumentation which is supported on lots of different architectures. This is the main reason for CELF's current project to define an [[Instrumentation API]]

*There doesn't seem to be a single API to support in-kernel timing instrumentation which is supported on lots of different architectures. This is the main reason for CELF's current project to define an [[Instrumentation API]]

Existing Instrumentation Systems

TimePegs

Printk Times

Produces printk's with extra time data on them. As of kernel 2.6.11 this is part of the mainline kernel enabled by CONFIG_PRINTK_TIME. Previous versions can add it via a very simple patch. It works for bootup time measurements, or other places where you can just jam in a printk or two.

Boot Tracer

Starting from 2.6.28 the kernel has this new feature to optimize the boot time. It records the timings of the initcalls. Its aim is to be parsed by the scripts/bootgraph.pl tool to produce graphics about boot inefficiencies, giving a visual representation of the delays during initcalls. Users need to boot with the "initcall_debug" and "printk.time=1" parameters, and run "dmesg | perl scripts/bootgraph.pl > output.svg" to generate the final data.

Kernel Function Instrumentation (KFI)

A system which uses a compiler flag to instrument most of the functions in the kernel. Timing data is recorded at each function entry and exit. The data can be extracted and displayed later with a command-line program.

Function trace in KDB

ftrace

Ftrace is a simple function tracer which initially came from the -rt patches but was mainlined in 2.6.27. Compiler profiling features are used to insert an instrumentation call (with gcc -pg option) that can be overwritten with a NOP sequence to ensure overhead is minimal with tracing disabled (this is enabled through CONFIG_DYNAMIC_FTRACE). There are a number of tracers in the kernel that use ftrace to trace high level events such as irq enabling/disabling preemption enabling/disabling, scheduler events and branch profiling.

Steven Rostedt, ftrace's main developer, gave a conference on ftrace at the Ottawa Linux Symposium 2008. The video and slides.

SystemTap / Kprobes

SystemTap is a sophisticated kernel instrumentation tool that can be scripted with its own language to gather information about a running kernel. It uses the Kprobes infrastructure to implement it's tracing.

Notes

Some random thoughts on instrumentation:

Most instrumentation systems need lots of memory to buffer the data produced

Some instrumentation systems support filters or triggers to allow for better control over the information saved

instrumentation systems tend to introduce overhead or otherwise interfere with the thing they are measuring

instrumentation systems tend to pollute the cache lines for the processor

There doesn't seem to be a single API to support in-kernel timing instrumentation which is supported on lots of different architectures. This is the main reason for CELF's current project to define an Instrumentation API