Introduction

One of the nightmares of a programmer is having their program's memory leaked. This is an important and noticeable concern in this crazy world of corporate politics, where your program may undergo several reviews, just to prove that your program is a buggy one (code reviews don’t focus on QA!!). Irrespective of the size of the application, it is very common for a programmer to make some mistakes. This article is for those who want their programs to be memory leak free.

Why you need to go through this

There are many tools that are available, and the debuggers can detect the memory leaks. Then why should you go through this article?? Debuggers don’t give the detailed output, and this is the first step to create a full fledged tool. And one more point that motivated me to write this article is, I have seen some postings in MSDN forums asking ways to detect memory leaks. This is currently in proc version, to make use of this you have to include the files available as download with this article. This is the first article in this series, and might span to 3-4 articles based on users feed back.

Brief introduction to heap memory

You might already be aware of the fact that, heap is a chunk of memory from which your program requests and releases memory on the fly. Windows heap manager processes the requests made by your program. Windows offers a variety of functions to deal with the heap, and supports compaction and reallocation. I am not going to discuss advanced heap memory management, and I reserve it for my future articles. This article concentrates on hooking the heap allocations, reallocations, and de-allocations.

Win32 systems support a logical address space of 4 GB, out of which 2 GB is reserved for OS itself, and the remaining 2 GB is left for the user programs. If your application requires more than 2 GB address space, you can request the OS to make room for one more GB, so that the OS adjusts itself to 1 GB, and allots the remaining 3 GB for your program to meet your requirements. Your program can run in a maximum of 3 GB address space, and needs little physical memory to support it.

By default your program's heap memory reserves 1 MB (256 pages of size 4 KB), and commits 4 KB (1 page). Whenever your program requests for some more memory, the heap manager tries to process your request by allocating it from the committed 4 KB, if it crosses the 4 KB boundary it will commit one more page. If your application requests for memory more than the default 1 MB, then it will reserve one more MB. Heap manager keeps this process until your program runs out of address space.

In the days of WIN16, Windows maintained two kinds of heap memory, one is global heap, and the other is local heap. Each Win16 application has one global heap, and one local heap. Applications are free to request the chunk of memory from any heap. But WINNT removed the concept of global and local heap, and introduced the new concept of one default heap, and a number of dynamic heaps. But WINNT still supports the Win16 heap functions like GlobalAlloc, LocalAlloc for backward compatibility. If your application requests the Win16 heap functions, WINNT maps them to the default heap.

By default your application owns a default heap, and you can create as many number of dynamic heaps as your applications needs. (I think there is some limitation on the number of handles like 65,535, but I am not sure!!!) Default Heap is the area of memory specific to the process. Usually you don’t need to get the handle, but you can get the handle by using GetProcessHeap(). Usual malloc, new calls will map to the default heap. Dynamic heap is the area of memory which can be created and destroyed by your application at runtime. There are some set of functions like HeapCreate, AllocHeap to work with the dynamic heaps. I will give you a detailed description on heap memory management in my later article.

This article uses the CRT diagnostics functions available as part of MS Visual Studio. Whenever I call the memory, treat it as Heap memory, don’t confuse it with primary memory.

About the debug heap

Debug heap is the extension to the base heap. It provides powerful ways to manage your heap allocations in debug builds. You can track any kind of problem from detecting memory leaks to validating and checking for the buffer overruns. When your application allocates memory by using malloc () or new, it will actually be mapped to its debug equivalents like _malloc_dbg(). These debug heap functions further rely on the base heap functions to process the user request.

Debug heap maintains a variety of information to keep track of the memory allocations. Suppose when you request 20 bytes, the debug heap functions actually allocate more memory than you requested. Then this extra memory will be used by the debug heap functions to perform some validation checks and bookkeeping. The debug header is stored in a structure _CrtMemBlockHeader, defined in dbgint.h file.

This header will be maintained as an ordered linked list. The first two parameters point to next and previous blocks:

szFileName holds the filename that requested the memory.

nLine holds the line number in the file, where memory requests happen.

nDataSize holds the size of the block request.

nBlockUse indicates the type of the block requested. This may be one of the following:

_CRT_BLOCK: CRT (C Runtime) functions make use of this type of blocks for internal use.

_NORMAL_BLOCK: This is a general type of block, allocated when requested using malloc or new.

_CLIENT_BLOCK: To keep track of a certain type of allocation you can call this type of blocks. It includes the subtypes. MFC internally uses this type of blocks to allocate memory for the CObject derived classes. Refer MSDN for more information.

_FREE_BLOCK: Blocks that are freed. You can keep the free blocks still in the list to check for threshold memory requirements, by using the _CRTDBG_DELAY_FREE_MEM_DF flag. These blocks will be filled with 0xDD to keep track of free blocks.

_IGNORE_BLOCK: When you turnoff the debug heap for a certain period of time, allocations that are recorded during this time will be marked as ignore blocks.

lRequest holds the block number (block numbers are assigned in the order of request) in the list.

Data holds the actual data.

gap and anotherGap surround the actual data. This is a 4 byte area (Microsoft calls it NoMansland buffer) filled with a known value (0xFD) to identify the buffer overruns.

So, whenever you request for some memory in the debug versions, you are actually allocated with some extra memory for bookkeeping information.

Detecting memory leaks by taking memory snapshots

_CrtMemState can be used to hold the memory state. When we call _CrtMemCheckpoint by passing _CrtMemState variable as parameter, it fills the state of the memory at that point. The following code snippet shows how to set the checkpoint:

_CrtMemState memstate1 ;
_CrtMemCheckpoint(&memstate) ;

You can find the memory leaks by comparing the different check points. Usually you need to take the first checkpoint at the start of the program and next checkpoint at the end of the program, and by comparing the two checkpoints, you can get the memory leak information. Like:

CrtMemState memstate1, memstate2, memstate3 ;
_CrtMemCheckpoint(&memstate1) // call at the start of your program
.............
............
_CrtMemCheckpoint(&memstate2) // call at the end of the function

Use the function _CrtMemDiffernce() to find the memory leak. Its syntax is as follows:

Detecting memory leaks by using hooks

This is the procedure that I followed to keep track of the allocations and deallocations. CRT debug offers functions to hook the allocations. When you call the hook function by passing it the pointer of your own function handler, it will be called whenever your program requests and releases the memory.

Here, nAllocType indicates the type of operation. It can be either of the following:

_HOOK_ALLOC - for allocations.

_HOOK_REALLOC – for reallocations.

_HOOK_FREE - for free requests.

userData is the header of type _CrtMemBlockHeader. It is valid for free requests and holds NULL value for allocation requests.

size holds the number of bytes requested.

nBlockType indicates the type of block (like _NORMAL_BLOCK). For the _CRT_BLOCK allocations, you better return the function with the return value true, otherwise you may get struck in the loop. You better don’t handle _CRT_BLOCK.

requestNumber holds the block number.

Filename is the name of the file that sent the request.

lineNumber is the line number in the above file, to pinpoint where the request happens.

You can replace the CRT functions with your versions for the tracking of memory management. But I used the hook functions only for logging purposes. Whenever allocation requests come, I fill that information in the ordered linked list, and remove the entry from the list whenever the corresponding free request is called. Here the block number is the main variable to map the free requests to the entries in our own linked list.

Using the code

Two sample files (MLFDef.h and MLFDef.cpp) are available as downloads with this article. To make use of these files do the following:

Add the files to your project.

Include the MLFDef.h in the file where you want to call the MLF functions.

Call EnableMLF() to enable the MLF (memory leak founder). It is better to call at the start of the program (like InitInstance()).

Call CloseMLF() to create the log file. The log file will be created in the C: drive. Its hard coded path is c:\MLFLog.txt.

Demo version is available to illustrate how to use the files.

Limitations

I experienced some problems, when used across the ActiveX modules. It is better to include the EnableMLF() and CloseMLF() in the main module until I update this article. It doesn’t work for VC7 but it works fine for VC6. I will update this article to include the support for VC7 and to fix the ActiveX module problems.

Extensions

This is just the in proc version, the first step to build a full fledged tool. You are encouraged to develop your own. This article is also the first part of the series, and you can expect some more articles on advanced memory management. You are suggested to go through MSDN for further information.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

Share

About the Author

I started programming in 1998, and bitten by the corporate bug in 2003, and currently working for a big shot in telecom domain. Started career as system programmer, but for better or worse shifted to application domain. Looking for good companion to share the thoughts

I get the same error and have not been able to find the problem. I get "fatal error C1083: Cannot open include file: 'dbgint.h': No such file or directory", which means I cannot use pHdr or _CrtMemBlockHeader, which I'd like to use. Any ideas? I'm using VS 2008 on Vista.

The vintage of this article dictates the tools that can be used to duplicate the situation available at the time of its development.

Remember that we are in the future now compared with when this article was written and the code developed. Microsoft changes its stuff on a frequent basis mainly to encourage product obsolescence, but a little bit to improve features and a little to decrease performance so that you have to buy a better, faster, improved PC every few years to run the new software properly.

I may be wrong... The architecture of MFC was deprecated to encourage dot Net. Instead of modern VB, and VC++ generating machine code, the compilers generate some kind of intermediate common code--CLI. It lets C# happen and may permit garbage collection schemes that may compete with that of Sun's Java JIT stuff. The terminology I encountered while Google searching this involved "execution engine" i.e. virtual machine. These are reminiscent of byte code interpreters (slow execution of pseudo instructions) or just-in-time compilers that convert the byte code into machine code when the OS loads the program for execution.

The VS debugging architecture has probably evolved to keep up with these new "advances" and so must use alternative techniques to debug dynamically allocated object memory problems?

If your code is pretty simple, it might be backwards compatible with the older development tools and libraries used here in this project so that you could install all of the older development tools, recompile, et voila--working debugging environment for you and your code, otherwise, your code may tie into newly released OS object interfaces / DLL library code that were not available in the 1998 to 2003 time frame so that your code would not compile / link / run in that older environment?

You can find these files in the C:\Program Files (x86)\Microsoft Visual Studio X\VC\crt\src directory, where X is the version of Visual Studio you're using.
_CrtMemBlockHeader definition can be found inside dbgint.h, as for the other functions mentioned in this article are implemented in the file dbgheap.c which is located in the same directory.
Hope it helps future readers.

I read your article looking for some reference to this problem I ran into. As I couldn't find this issue documented anywhere else, I'll post it here for future reference.

A CrtMemState checkpoint records the address of the most recently allocated heap block. CrtMemDumpAllObjectsSince walks backwards through the heap, starting with the most recently allocated block, until it gets to either the checkpoint block, or the start of the heap.

In a dynamic system, this frequently causes CrtMemDumpAllObjectsSince to dump the entire heap. This occurs because the block that happened to be allocated just before the checkpoint was created was freed before the dump occurred.

I worked around this problem with the following code which I use when I need to create a checkpoint.

I like your discussion on the background of Windows Heap Memory... but the discussion on CRT usage is too advance for me. If you don't mind, I would like to bring your attention to another article. This article discusses: how to overload the new and delete operators to find C++ memory leaks.

there are many ways to detect memory leaks.
The basic motive behind this article series is to replace the default debug heap.
This article is just to introduce the debug heap to the devolopers( i have seen many postings in the forums), and in my next article i will replace the windows debug heap, and will provide some more functinality than the current one with my new debug heap. Thats why i added the code to show how to hook debug heap requests and maintaining the bookkeeping information. Wait for my next article, u will find some interesting stuff.

As much as you care about sharing your knowledge of, "Detecting memory leaks," through the use of CRT diagnostic functions, I believe a lot of the 'fluff' from your article could have been omitted (especially when the substance of what you've written comes from just two sources and could have easily been summarized).

The first source, readers will find on pages 656 and 657 of Jeff Richter's book, "Programming Applications for Microsoft Windows: 4th Edition."

The second source is at this MSDN address, http://msdn.microsoft.com/library/en-us/vccore98/HTML/_core_memory_management_and_the_debug_heap.asp

I offer this observation because if your proposed future articles are going to be based on what Richter has written on pages 657 through 663, it might help if you were to write a draft copy first, followed by several edited versions.

There is nothing wrong in wanting to share your knowledge with the community, just do it in a substantive way that reflects you know what you're talking about.

thanks for ur suggestion. Even now i didn't gp through jeff ritcher'k book. This article is based on MSDN, and the CRT source files supplied with VS.

what u think, how jeff ritcher or any other guy writes on any technology developed by others. U mean to say they wrote those books and articles by guessing with their super natural powers, or they followed the developers manual. This is purely based on MSDN, and i already told the puprose of the article, and future versions of the article eloborate the concept.

what about the articles that describes creating dialog boxes?? U can see thousands of books describing the creation of the dialog boxes in the same way, that does mean they followed all that books to write these articles.

William, get off your high horse. There is almost nothing on this entire site that cannot be gleaned from a respected body of work somewhere. What the contributors do is draw the attention of the site user to coding practices in a useful and pragmatic way.

For myself I am looking into detecting and reporting memory leaks via an automation interface for some COM modules, which is how I arrived here, I actually own Jeff Richter's book, and I had no idea that page 656 onward contained anything relevant - it's a vast book and I certainly have not scoured every page (Shame on me - but I do have a life to lead as well).

The posted article and code have been beneficial in achieving my original goal, and even your pompous comments were not wasted as they have also helped in pointing me to new reading material on the subject.

I write this as an encouragement to narendra bommineni to carry on submitting.

For something to be useful and pragmatic, does not mean readers should have to suffer through fluffs in order to gain from its value.

I have nothing against the value of what this person sought to contribute, but if your time is valuable to you, then you shouldn't have to find yourself repeat-reading certain sentences and passages in order to extract the substance of what is written.

It is only because I'm very familiar with the writings of Jeff Richter that I knew what this person was talking about, but in the absence of such familiarity, readers would have no choice except to suffer through the excesses and poorly formed contents.

Shouldn't the idea then that the author first come up with a draft copy of his material, followed by several edited and improved versions afterwards, make sense? That's exactly what I offered!!

Nobody is saying this person should write like Jeff Richter, but I am certain what Richter has written (and published), WEREN'T DRAFT COPIES of his own works either.

When you see some link on CRT heap, what u will expect from the link.
With ur comments i checked jeff ritcher's book. There is a lot diff between books and articles. You can't explain all the function signatures(that's what jeff did in his book) in the article, then it will become book.

Refer "windows 95 system programming secrets by matt pietrek", most of the book filled with data structures explanation, but that book is one of the premier in OS internals.

When u see some stuff that u already know, then it seems as a fluff to u. That is human psychology. I also gone through many articles in this site regarding coloring controls like edit, grid. Becuase i already aware of that thing, i couldn't say fluff.

Those two files that i included, have written for my personnel use. I dont have any thought to post articles of this sort. I got this idea, when i have seen some postings in MSDN forums. Then i included those two files, and prepared this article in not more than 3 hrs, and planned to elaborate it.

can u send ur blog address. Just to know how to write articles by seeing ur stuff, that will helpful to my future ones.