Using sp-rtrace

The sp-rtrace tool allows you to run programs, and collect resource allocation/free backtraces from a given set of functions, even from optimised binaries. The sp-rtrace tool supports plugins that define the set of functions that the tool traces. Multiple plugins can be used at the same time, but trace toggling affects all of them.

Packages

source: sp-rtrace

binary: sp-rtrace, sp-rtrace-visualize

Installing sp-rtrace on the Harmattan device

Install sp-rtrace through the developer mode applet.

Prerequisite: Developer mode must be enabled.

Select Settings > Security > Developer mode.

Install the Tracing bundle package by clicking Install.

You get a notification screen that lists all the applications to be installed in the bundle package. To install, click OK.

Features

sp-rtrace: command line front end which outputs/saves the trace data as binary.

libsp-rtrace-main: with the plugin LD_PRELOAD libraries, they implement the tracing functionality and transfer the traced data to the front end process.

rtrace-module-gen: included with the sources for generating trace modules from standard templated and traced functions configuration file.

libsp-rtrace1: API library for marking when a given process context starts and ends (so that, for example, servers can differentiate resources specific to each client).

sp-rtrace-postproc: converts the binary trace data into ASCII, and optionally:

Merges identical backtraces in the trace data.

Removes allocations for already freed resources.

Calls sp-rtrace-resolve (with -t elf option).

sp-rtrace-resolve: fully resolves the symbols (using separate debug symbol files) for the backtrace addresses. Compared to Valgrind, the symbol resolving capacity of the sp-rtrace tool is equal or better. The sp-rtrace tool can use both ELF- and BFD-based resolving.

Several utility scripts for getting trace statistics and so forth, see the following sections.

When to use sp-rtrace

The sp-memusage, sp-endurance, and sp-smaps tools give you the system view on the process memory usage. This is aligned to page size (4KB) and due to memory fragmentation, the process heap usage can sometimes actually be much larger than what the process itself needs. The sp-trace tool has the rtrace-allocmap post-processing tool to visualize (memory) fragmentation.

Valgrind Memcheck plugin can detect lost pointers to allocations, so with Valgrind you do not need to repeat use cases to see whether something leaks. Sp-rtrace (and functracer) are intended for analysing non-freed allocations and for the ones which still have valid references sp-rtace, with its post-processing tools, is better than Valgrind. Before performing memory leak analysis with sp-rtrace (and functracer), fix all the memory leaks reported by Valgrind. With sp-rtrace you usually need to repeat your test cases several times and do this separately for each test case.

Valgrind Massif plugin shows process memory usage as a function of time (with backtraces) over whole application lifetime whereas sp-rtrace can take a snapshot between any given points in time and do call graph of allocations done during that time interval. Both are useful, and it is recommended to use both (they may show different things). Sp-rtrace is faster than Valgrind as it does not emulate the CPU.

Functracer is similar to sp-rtrace, the main difference is that it can attach to an already running processes and can catch some things that sp-rtrace (LD_PRELOAD) does not.

Using the tool

Note: To use this tool with the developer account, run the sp-rtrace tool with the aegis-exec command. For example:

To additionally make backtraces (-b) eight frames long and store the output to "trace" directory (-o), enter the following command:

mkdir trace
sp-rtrace -s -e memory -b 8 -o trace/ -x ls

Toggling tracing

If you want to trace some specific use case instead of process startup, just leave the -s option out and toggle tracing at runtime. Enter the following command:

sp-rtrace -t $(pidof program)

To do the same directly, enter the following command:

kill -SIGUSR1 $(pidof my-application)

You can also use the latter option in the program itself if you want tracing to start from a specific place.

Post-processing

Trace files are by default in binary format and named PID[-index].rtrace. Post-processed files are in text format and named PID[-index].rtrace.txt.

When investigating more complicated programs, merge identical backtraces and remove freed allocations. To do this, enter the following command:

cd trace
sp-rtrace-postproc -cl < 14422.rtrace > 14422.rtrace.txt

You can use sp-rtrace-resolve to resolve the remaining backtrace addresses. Enter the following command:

sp-rtrace-resolve < 14422.rtrace.txt > trace/14422-r.rtrace.txt

By default, this uses the slow and memory consuming BFD resolver to get the source file names, line numbers and names of non-exported functions. If you are running it on the device, use also the -m multi-pass option to reduce the memory usage after making sure that the current directory has a lot of free disk space for temporary files.

Note that the relevant debug symbol packages need to be installed before the backtrace addresses can be resolved. Additional debug symbols slow down resolving and increase the amount of memory needed, but especially for call graphs it may be better to have all the function names, as nodes without names are not included into call graphs.

To make sp-rtrace-postproc run the resolver automatically, use the -r option. Enter the following command:

sp-rtrace-postproc -clr < 14422.rtrace > 14422.rtrace.txt

Note that the command asks sp-rtrace-resolve to use the faster ELF resolver which also uses less memory, but it does not provide more detailed information, such as line numbers.

Note: With the -P option, you can ask sp-rtrace itself to do all of the above post-processing when saving the trace output. Enter the following command:

sp-rtrace -s -m -e memory -o trace/ -P "-clr" -x ls

You can do all the post-processing in one step, unless you need separate text data files for the following:

all the resource allocations as required by the allocation timelines generation

only non-freed resource allocations as required by the call graph generation

If ELF resolving referred above does not give enough information, use sp-rtrace-resolve explicitly. Enter the following command: