FreeBSD recently grew support for profiling in many different ways through
use of the hardware's performance monitoring counters (PMCs). This lets you
sample things like instructions retired, cache hits/misses, branch mispredicts,
etc. either for a single program (and its descendents) or the entire system.

This has been a great aid for me while working on graphics performance in
FreeBSD. The usual problem here is that some command is slow, and I want to
know what's responsible, whether it's cairo, liboil, X.Org, the DRM, etc., and
I want to know what functions are responsible within those pieces of software.
The pmccontrol/pmcstat programs will give me a pile of .gmon files of profiles for each of
these when I run a command under them. So, I made a set of simple perl/sh
scripts for personal use to post-process these .gmon files into a nice flat
output for programs I use.

These scripts are very under-developed. We really need someone to write a
unified program to handle hwpmc results, but for now I've managed to become
much more productive using just this.

runpmc: This is what I use to run commands and
sample instructions while doing so. It calls procgmon.pl to post-process the
.gmon files produced and hand me a summary. Note that you will have to
change the k8-fr-retired-x86-instructions line to the appropriate
directory for your machine and sampling mode. Usage: runpmc
command [args...]

procgmon.pl: This perl script iterates over the
.gmon files in the current directory, trying to match them to binaries, then
running gprof to produce a flat profile and post-process it into straightforward
output per function showing number of ticks in the function and the cumulative
(over all binaries) percentage of CPU time spent in the function. Usage:
procgmon.pl

If you get a complaint when using these scripts (or sampling mode in general)
while using a UP i386 about Operation not supported, this is a limitation
of hwpmc at the moment, according to its author. The problem is that many UP
i386 motherboards disable the local apic, which is required for hwpmc to deliver
the interrupts how it wants to.

If you get a crash at the end of the pmccontrol run, this may be a current
issue that jkoshy is working on. For now, apply this patch:
hwpmc-logging-dontpanic.diff (from jkoshy)