Debugging memory leaks and memory corruption problems is
something of a black art; there is probably no one right way and no one right
tool to tackle these problems, most often you will have to try out multiple
approaches and multiple tools.

The DGBMEM toolset is a drop-in memory allocator for Linux
that adds some debugging functionality, it overrides the standard library
functions like malloc and free that are returning new heap memory and adds some
extra debugging functionality/

In order to use this tool you can use your process as is,
and you don't have link your process with any libraries. This works well for
ELF executables.

DBGMEM is a relatively primitive tool if compared with other
tools like valgrind; though DBGMEM may sometimes be better suited for testing
systems under live condition and under heavy load.

The tools also helped me to solve problem with real world
systems, in other words it helped to save my but with a real job - which I
think is very much for a tool like this ;-)

DBGMEM is a very feature-rich tool (some say that this is a
euphemism for bloat, but well, Features Are a Good Thing).

A report containing all memory blocks + stack trace at
allocation time is produced at program exit, or by application request; at
the same occasion all memory blocks are checked for
overwrites/underwrites.

Your application can call service functions that checks
all memory blocks for overwrites / underwrites, or to print out all heap
memory blocks; alternatively the tool enables you can install two signal
handler for each of these task.

Whenever a memory error is encountered the event is logged
into a log file and to standard error stream; you have the option to

Continue as normal

Dump core

Attach the gdb debugger to the currently running process.

The tool hooks the following GLIBC functions, and track
dynamically allocated memory that is returned by them

malloc

calloc

realloc

memalign

posix_memalign

valign

free

strdup

strndup

getcwd

new/new[]/delete/delete[] (for C++ version only)

malloc_usable_size returns size of requested block.

There is an option to initialize all allocated heap memory
with 0xDD and to set all freed memory content to 0xFF.This is a strong
feature, references to freed memory or memory that is not initialized will
cause program errors, which are otherwise left undetected. If
freed/un-initialized memory contains a pointer, then dereferencing of this
pointer results in a core dump.

There is an option to install SIGSEGV/SIGBUS signal
handler that also checks all heap blocks for overwrites.

You can run an option that will invoke the top program in
parallel with the debugged program so that it exclusively monitors the
memory consumption of the debugged process.

pinpoint and track memory leaks in live systems with
minimal overhead;

provides some checks for heap corruption; this tool keeps
additional housekeeping information in a header before the allocated block
(arena header) and after the allocated block, so heap corruption check is
no always perfect; the checking tool supports these situations better.

efficiently check for heap corruption, pinpoint and track
memory leaks in live systems Housekeeping information is kept off the heap
in an anonymous shared memory segment, so the heap checking is even better
here, though a bit slower.

Checks for free of memory block that was not allocated.

For a set of standard library function we also check for
following errors

heap overwrites in protected set of functions

Reading past allocated heap memory range

Stack smashing (limited support, the check triggers only
if you smash the function return address)

When a memory error occurs then the event is traced to
standard error file;

A report is generated after running the debugged process
(see file dbglog.log)

The report consists of the following sections

information returned by mallinfo (man page 3) information
at the time that the report has been generated

memory errors; memory overwrites, twice free, etc. etc.

leak reports

A leak report can be divided into 'generations', i.e.
distinct phases of the program execution can be tagged (say the initialization/setup
phase is generation 0 and everything after setup is phase 1 - see ‘Memory
Leaks section’ for more information)

Memory leaks cause the process to consume more and more
memory, leading to general performance degradation; swapping, and eventually to
the process termination due to the fact that the process has run out of free
memory. DBGMEM can help you to find these issues effectively.

The lifetime of a process can be divided into following logical
stages:

[---
INITIALIZATION ---]

[---
ACTIVE STAGE ---]

[---
SHUTDOWN ---]

[---
EXIT ---]

Initialization

The process is initializing. Usually here the configuration
is read, caches are initialized and objects and resources used throughout
further stages are created and initialized.

Active state

The process is servicing request and does something useful;

During this stage the process completes one or more logical
units of processing,

Each such unit may be the processing of a request or series
of requests from a network, the processing of a batch data job, or the
processing of an interactive user requests. Typically during each logical unit
of processing some resources and memory is allocated and then either released
or leaked.

A leak is a resource that is allocated and not freed during
ACTIVE STAGE of a process. If you allocated a block during initialization and
forgot to clean it up then this has a fixed cost and will not have any affect
(unless you have a shared memory space between processes).

.

There are the following types of leaks

Simply leaked; it is allocated and then we simply forgot
to free it.

Added to caches for further use; this pattern may lead to
a leak if an object is always added to a cache, but is never reused from
that cache (bad cache leaks).

Leaks related to reference counting; Reference counting is
a special form of garbage collection that is often implemented in C/C++
programs; there are potential problems such as

Leaking of reference counts; a reference counted object
is only freed if all outstanding references are released. A leak can
occur when at least one reference to an object remains, because somehow
we forgot to clear an outstanding object reference. This scenario often
occurs if a root object refers to all other objects but forgets to clear
its references. Note that this is similar to bad cache leaks. Note that
this scenario often happens in other programming languages such as Java.

Circular references; object A is referencing object B,
which is also back-referencing object A; No matter what, the object A will
always have at least one back reference from object A, so it will never
be deleted.

The DBGMEM tool creates a detailed memory report that lists
all memory blocks present at a particular point in time, this report includes
the stack where each block was allocated; all heap objects that were allocated
from the same stack location are summed up in one report entry.

Two reports taken at different points in time contain
interesting information;

·On completing the SHUTDOWN sequence; right before
the process exits normally This report is always generated by DBGMEM unless the
process is killed by SIGKILL signal; you don't need to modify you program in
order to get this report. This report lists the origin of most leak issues; the
stack trace where the leaking memory blocks were allocated will give you a
strong hint as to how to fix these issues, alas for some types of leaks related
to reference counting you need more information.

·On completing the ACTIVE STAGE right before the SHUTDOWN
sequence

We want to tag each memory
allocation with the stage in the program lifetime when it was made. For
instance we want to know if an allocation was done during INITIALIZATION stage,
so we will not have to deal with it at all.

This stage is useful if you want
to track reference counted leaks; usually these leaked references are removed
during SHUTDOWN stage, when the root objects that are holding these references
are deleted. When a lot of blocks occur in the report taken before SHUTDOWN
stage, but are not present in the report taken before EXIT then this can
indicate a leak. This analysis is not done by the tool, it is done by the avid
reader, when he compares the two reports; the avid reader is also armed with
knowledge of the workings of his program and will be able to conclude what is
happening.

There are two ways to indicate
the two state transitions

1.INITIALIZATION and ACTIVE STAGE

2.ACTIVE STAGE and SHUTDOWN stage

You can either

1.Enable DBGMEM command line option that sets up two signal handler,
where each signal handler will do the requested state transition.

2.Modify the debugged program so that it calls the debug library to
indicate the two events; this is the preferred solution if you want to run
DBGMEM from within unit tests/system tests.

The tool is a drop in replacement for part of libc/runtime
library functionality; you don't have to link your application against a debug
library, instead this tool is implemented as a shared library that is loaded before
loading the shared library of the runtime library by means of manipulating the
LD_PRELOAD environment variable.

The tool consists of the following components

/usr/mdbg/scripts/run script, the script that 'drives' the
debugging process. It does the following steps:

Parses the command line.

Writes file with command line parameters, first trys
/tmp/dbgmem_param_$pid then /var/tmp/dbgmem_param_$pid then
~/dbgmem_param_$pid ($pid of run script); We can’t use environment
variables to pass parameters to debugged process, at the library initialization
(_init) time there still is no environment.

Libraries /usr/local/dbgmem/lib/libdmemc.so
/usr/mdbg/lib/libdmems.so - each shared library is one drop in memory
debugging tool. It gets a first pass on any memory allocation function,
so it adds some housekeeping information to each allocated block. The tool
creates raw report files that are further processed after the debugged
process has terminated. At program exit, or when requested the following
set of files is produced

DBGMEM_report.log a raw report that lists each
memory allocated by malloc with stack where it was allocated,

DBGMEM_report_mmap.log a raw report that lists each memory
allocated by mmap/mremap with stack where it was allocated,

File maps a copy of the file that
lists memory map of debugged process

Each tool is initialized by _init
method of shared library.

Library /usr/local/dbgmem libdmems.so implements the
simple tool which adds arena header to each allocation which

Next/previous block pointer

Stack of when block was allocate

Size of block

Generation tag value + kind of allocation function
(malloc/new/new[])

After information block, and after end of user allocated
block a sizeof(void*) guard area is keps, it is initialized by sentinel
values so that we can check for memory overwrites / underwrites.

This arena header can be
overwritten, although things are not quite as bad, since we can double check
the linked list as it is traversed; and we are not advancing in it if an entry
has gone wrong; bigger problem is that the stack trace can get overwritten.

Library /usr/local/dbgmem/lib/libdmems.so implements the
checking tool that is a bit more complicated.

Keeps a radix tree that maps the pointer value to
information block

Information block keeps

Next / previous information block

stack trace where block was allocated,

Generation tag value + kind of allocation function
(malloc/new/new[])

size of allocated area,

Pointer to start of memory region

After information block, and after end of user allocated
block a sizeof(void*) guard area is maintained, it is initialized by
sentinel values so that we can check for memory overwrites / underwrites.

Radix tree and information blocks are kept off the heap;
they are allocated in anonymous shared memory regions. Allocation within
that region is fast, since we are using fixed sized block allocator.

Since radix tree keys are ordered, we can find the
beginning of a memory block for any pointer contained within that block;

For any pointer arguments that is passed to functions as
strcpy/memcpy we find the information block that describes the region
that contains the argument pointer, now we can check if the strcpy/memcpy
function attempts to overwrite or read passed a heap range.

After the debugged process exits the annotate.pl script is
started by run script. Script /usr/mdbg/scripts/annotate.pl reads the raw
reports and creates the final report that

Turns each address from the raw report into a function
name; we use gdb to do this, if no symbol is found, then we can at least find
the name of shared library where the stack frame was; this is done my
looking at the maps file.

Unites memory block reports with identical stack into the
same report entry.