Introduction

Microsoft Visual C++ versions 5 and 6 sport an automation interface intended to enable developers to extend the product. Unfortunately, the interface is maddeningly shallow and buggy, making it near impossible to write addins that perform more than just simple tasks.

Some third-party products, such as Bounds Checker, integrate themselves seamlessly with the Visual C++ IDE. However, these products are not addins, nor are they limited by the miniscule capabilities of the automation interface. Instead, these products are implemented as MFC extension DLL's (but have a .pkg file extension). The developers of these products are privy to an undocumented interface of the Visual C++ product, namely the DEVSHL extension library, which implements the core of the Visual C++ application. DEVSHL.DLL is an integral part of Visual C++, and its exported methods are available by ordinal only (no method names). To use DEVSHL.DLL, one would require corresponding .h and .lib files. Whether the developers of products like Bounds Checker obtained these files from Microsoft, or engineered them from scratch, I do not know. If someone were to send me a DEVSHL.H and DEVSHL.LIB file, I would be forever in their debt.

Some shareware and commercial addins have achieved integration with Visual C++ that is seemingly more advanced than what the automation interfaces would allow - without any inside knowledge from Microsoft. Examples include Oz Solomonovich's WndTabs addin (here and here), my own WorkspaceEx product, and RadVC. Not only do these addins provide strikingly complex new features to the user, they also integrate cleanly with the Visual C++ user interface. The intent of this article is to provide insight as to how such integration may be achieved.

This article will sometimes refer to my WorkspaceEx product, which is shareware and available from my CoDeveloper website here [Editors note: this link is no longer active]. Although I don't distribute source code for that product, in this article I will describe the majority of techniques used by WorkspaceEx to perform its integration with Visual C++.

Visual C++ is a MFC Application

This should come as no surprise. To verify that this is true, simply look at the dependency list for MSDEV.EXE. It is bound (pun intended) to load MFC42.DLL.

In fact, MSDEV.EXE is a prime example of what a typical large MFC application is like. If you could peer inside at the code (we'll do that in a moment), you would see that Visual C++ makes heavy use of the document/view architecture, derives the majority of its classes from CObject, and leverages core MFC classes like CString and CObList.

One interesting point you would likely note is that many features in the IDE are implemented "the hard way". For example, Visual C++ uses a tab-control metaphor throughout the product. However, none of its tab control widgets are the native Windows common tab control. Instead, they are a custom class implemented specifically for Visual C++. In guessing why this is, I would speculate that the IDE was written before the tab control was finished or available on Windows 95, or that that control was not available for Windows NT 3.5. (My own personal history of Windows knowledge pretty much begins with NT 4.0).

The final giveaway clue that Visual C++ is a MFC application is that developers are invited to write MFC extension DLL's for the IDE to load in the form of custom App Wizards.

Package (.PKG) Files

Visual C++ is implemented as an executable (MSDEV.EXE) and several MFC extension DLLs. The majority of these DLL's have a .PKG file extension, and these live in the BIN\IDE subdirectory. Visual C++ is written such that it will load at startup any .PKG file it finds in this directory. Microsoft ships additional features with the Enterprise version of the product in the form of .PKG files.

Although .PKG files are typically MFC extension DLL's, this is not a requirement. Rather, a .PKG file need only be a DLL that export two functions, InitPackage() and ExitPackage(). These methods appear to have the following prototypes:

Visual Studio will load the .PKG file and call the InitPackage() method as part of its initialization process. .PKG files should return 1 to designate a successful initialization. Likewise, ExitPackage() is called as the IDE shuts down.

The parameter passed to these methods is the instance handle of the .PKG file itself.

For the benefit of readers who are a bit rusty on the specifics of DLL's that use MFC, allow me to refresh your memory. There are two basic types: "regular" DLLs have their own CWinApp object, and "extension" DLL's share the CWinApp of the calling application.

A .PKG file that is implemented as a MFC extension DLL has a distinct advantage. It has access to the CWinApp and other MFC objects owned by Visual C++.

The easiest way to create a custom .PKG file is to use the MFC AppWizard (dll), and choose the Extension DLL option. Then add and export the InitPackage() and ExitPackage() methods. Update the project settings and .DEF file to produce a .PKG filename instead of .DLL.

To gain access to the MFC objects inside Visual C++ from the InitPackage() method of a MFC extension .PKG, simply call AfxGetApp() and use the returned pointer as a jumping-off point.

It is important to note that MFC extension .PKG files must link against the release version of the MFC DLL runtime. More details on this requirement follow in the Better Automation Addins section.

Package files offer an alternative method to writing extension applications for Visual C++. The IDE will load .PKG files earlier than automation-based addins, and the extension need not be implemented using COM. However, a .PKG extension will not have direct access to the automation interfaces, if they are desired. For those of you who are interested in knowing, WorkspaceEx is currently implemented as both a .PKG extension and an automation addin.

Automation Addins

I would bet that 95% of the addins for Visual C++ begin life as code generated by the "DevStudio Addin Wizard". It is an unfortunate beginning, that is, to begin existence as such ugly crappy code. If your addin has its roots in such code, I highly recommend reading through it some time and consider its overhaul.

The code generated by the addin wizard compiles to a COM DLL that implements an object that will be instantiated by Visual C++. An interesting facet of this code is that codes heavily to both MFC and ATL, a technique not often used. The immediate question is "why", because the entire implementation could easily be coded entirely in ATL. In my mind, this would result in a much simpler implementation. (Likely, ATL was pretty "young" when this wizard was written - indeed it shows in the code that is generated.)

Visual C++ uses a highly suspect technique to identify and instantiate addin objects. The primary question here is how does the IDE know the CLSID or ProgID of the addin object itself, so that it may be instantiated?

Consider the steps taken to install and use a new addin. Typically, the user will open the Tools / Customize dialog and select the "Add-ins and Macro File" tab. Assuming the addin does not exist in the Addins subdirectory of the application install, the user will browse for the addin DLL. Once identified, Visual Studio loads the addin and makes it available to use.

This is all well and good, except for some mysterious magic employed by the IDE to determine what object to instantiate from the DLL, which may not even be registered when initially identified by the user.

An addin is simply a COM object with a unique CLSID. It must implement the known and documented IDSAddin interface so that when instantiated by Visual C++, Visual C++ can call known methods on the object, specifically OnConnection() and OnDisconnection(). It is the addin's responsibility to call back into the IDE during OnConnection() and pass the dispatch interface of (typically) a second object that implements the actual addin command set. Since the addin object CLSID is unique and not registered with the system in any way, how does Visual C++ know to create it?

I suspect that when Visual C++ identifies a potential addin DLL, it loads the DLL and calls DllRegisterServer(), a well known and required method found on all in-process COM servers. The addin DLL proceeds by making calls to the system to update its registration information into the system registry. I believe that the IDE watches these updates to discover the CLSID's of the objects being registered. It then creates these objects and queries them for the known IDSAddin interface.

An alternative to this improbable theory would be that Visual C++ loads the addin DLL's type library, and enumerates the coclass objects. However, I don't believe this to be the case as an addin will still work if the coclass entry for the addin object is not present in the type library.

A much simpler and cleaner architecture would have been for addins to register themselves as belonging to a specific component category. Visual C++ could then enumerate the objects of this category and instantiate them. But hey, that's just how I would have done it.

Regardless, once the addin is successfully loaded for the first time, Visual C++ adds it to a list of known addins in the registry. The registry key of this list is:

HKEY_CURRENT_USER\Software\Microsoft\DevStudio\6.0\AddIns

It is entirely possible for an addin to self-install itself by placing appropriate entries beneath this key. These entries consist of a sub key that is the ProgID of the addin, and it must have a default value of "1" to be loaded by the IDE. (This value corresponds to the check-state in the list of addins on the Addins and Macro Files tab.) Three required string values below this key are: Description, DisplayName, and Filename.

My own addins self install by making the necessary registry entries during the normal COM registration performed by DllRegisterServer(). This alleviates the necessity of writing separate code in my installation program, and is a generally clean solution. The installer simply registers the DLL, and the next time Visual C++ is launched, my addin is available.

Unfortunately, addins generated by the DevStudio wizard do not make use of .RGS (registration script) files, instead opting to make registry updates in code. I suggest overhauling existing such addins to use .RGS scripts, and using my own Addin Wizard (discussed below) for future addins. Use the features folks; don't reinvent the wheel.

Below is a sample .RGS script similar to the one employed by my WorkspaceEx addin. The first section performs the self-install; the second performs the object registration.

Better Automation Addins

The wizard-generated addin implements a "regular" MFC DLL, that is, it has its own CWinApp object. This approach precludes the addin from accessing the CWinApp object of the calling Visual C++ application, as is possible from a .PKG extension module.

It is possible to create an addin that is both an in-process COM server (to serve the Addin object itself), and a MFC extension DLL (enabling access to the guts of Visual C++). The caveat is that no Microsoft wizard exists to spit out such a beast. To remedy this situation, I have written such a wizard, and it accompanies this article.

"CoDeveloper's Extension Addin Wizard" is simple to use. Select the wizard, modify the default name and description of the addin, and generate code.

If someone wants to extend this wizard to generate .PKG files, or provide additional functionality, please be my guest. I'll gladly incorporate changes, so long as the code that gets generated isn't crap.

The CoDeveloper Wizard generates an updated version the code that the old Microsoft Wizard generates. The major differences are that the target DLL is a MFC extension DLL, and the COM objects register with .RGS scripts. In my opinion the code is much cleaner and maintainable.

There are a few caveats to be aware of when building MFC extension DLL's that are loaded by MSDEV.EXE. These apply both to the .PKG files detailed in the previous section, and the addins generated by the CoDeveloper wizard. Foremost, you cannot successfully link against a debug version of the MFC runtime DLL. This is because MSDEV.EXE will only ever load MFC42.DLL, and the debug MFC42D.DLL has a slightly different layout for its structures in memory.

My advice to overcome this potential headache is to immediately delete the Debug configuration from your addin project, and re-add it based on the Release configuration. Then add back the debug compile and link options; just don't add back the flags that denote linking to the debug runtime libraries (and don't #define _DEBUG). This Pseudo-debug build configuration will work just fine - you'll be able to debug to your hearts content, you just won't be able to trace into the runtime libraries. FYI, I was too lazy to make the CoDeveloper wizard generate a project with the Pseudo-debug configuration setup from the get go. I took one look at the pathetic means of specifying a new configuration via the automation interface and said to myself, "you've got better things to do."

Alternatively, you can rebuild MFC42.DLL to include debug information. However, I found that on my system, the makefile for MFC doesn't build correctly with VC++ service pack 4 installed.

Stupid Addin Tricks

You may be wondering why the heck I'm so keen on gaining access to the internal MFC objects of the IDE. To help answer this I've written an addin called OpenVC. OpenVC implements two commands: DeleteNCB and ShowInnards.

DeleteNCB is very similar to the NukeNCB command in my shareware product, WorkspaceEx. It enables the user to automatically destroy a workspace's corrupt .NCB file. .NCB files periodically become corrupt, and when they do, features like Intellisense and ClassView stop working. The solution has historically been to close the workspace, open a shell window, navigate to the project directory, delete the .NCB file, and finally reload the workspace. DeleteNCB and NukeNCB perform these events with the click of a button. DeleteNCB is slightly less convenient than NukeNCB because of a Visual C++ dialog box that pops up to ask the user if they want to close all open documents.

The problem with implementing DeleteNCB is that the addin needs to know the path to the .NCB file, so that it can delete it. Knowing the path to the workspace file (.DSW) would be sufficient, since the two files live in the same directory. However, while the automation interface provides means to enumerate the projects of a workspace, it does not provide attributes of the workspace itself.

DeleteNCB employs a nifty solution to this problem. A little experimentation reveals that the Workspace is implemented as a CDocument-derived object in the MSDEV class hierarchy. DeleteNCB simply enumerates the open CDocument instances until it finds the workspace. It then closes the workspace, deletes the .NCB file, and reopens the workspace.

I can hear you saying, "this is nice, but how did I know to look for the class CProjectWorkspaceDocTemplate?" The answer is a tool I wrote called ShowInnards. ShowInnards is the second command in the OpenVC addin, and its purpose is to map out the class hierarchy of Visual C++.

ShowInnards() works by using several techniques to discover CRuntimeClass information for the MFC classes in the Visual C++ process. Using ShowInnards, I've been able to identify over 200 classes, though I suspect there are likely a few more stragglers that could be identified with a little more work.

ShowInnards() identifies a class's name, size, derivation, and schema version (rarely used or relevant). It also attempts to discover the layout of the class instance in memory - that is, it attempts to determine a class's member variables. It does this by examining an instance of the class in memory, byte by byte. Using a bit of cunning, it determines whether a range of bytes is either a pointer to a CObject derived class, or an actual embedded CObject derived class. It does this by looking at the potential contained object's would-be virtual-function table. If the bytes it examines are consistent with a CObject virtual-function table, it calls the GetRuntimeClass() method of the identified object.

bool CDlgClasses::GetRuntimeInfo(void* pvObj, CRuntimeClass** pprc)
{
CObject* pObj = (CObject*)pvObj;
// We want to retrieve the MFC runtime type info pointer from this potential
// CObject pointer. The problem is, the pointer may not actually point to
// a CObject. If we just blindly cast and call pObj->GetRuntimeClass(),
// we're likely to try executing some non-code, or perhaps we'll call some
// other virtual function like a destructor.
// Since this would be bad, we attempt to check the signature of the function, to
// see if it "looks" like GetRuntimeClass(), which I've found to be simply a
// mov instruction followed by a ret instruction.
//// I won't guarantee that this code will always work. If it starts failing,
// you fix it.
__try // hope that SEH works, and hope we don't need it!
{
// validate address:
if (FALSE == AfxIsValidAddress(pObj, sizeof(CObject), FALSE))
returnfalse;
// check to make sure the VTable pointer is valid
void** vfptr = (void**)*(void**)pObj;
if (!AfxIsValidAddress(vfptr, sizeof(void*), FALSE))
returnfalse;
// validate the first vtable entry
void* pvtf0 = vfptr[0];
if (IsBadCodePtr((FARPROC)pvtf0))
returnfalse;
// look at the code for this function. validate it is a mov and ret
BYTE arrOpcodes[6];
memcpy(arrOpcodes, pvtf0, 6);
// prepare yourself, this gets ugly.
// if you don't understand whats going on then leave well enough alone.
// don't get all upset about it either.
if (arrOpcodes[0] == 0xFF && arrOpcodes[1] == 0x25) // jmp
{
void** pvAddr = *(void***)&(arrOpcodes[2]);
if (IsBadCodePtr((FARPROC)*pvAddr))
returnfalse;
memcpy(arrOpcodes, *pvAddr, 6);
}
if (arrOpcodes[0] != 0xB8 || arrOpcodes[5] != 0xC3) // mov, ret
returnfalse;
// ok, it looks like a likely candiate for a "real" GetRuntimeClass().
// go ahead and call it.
*pprc = pObj->GetRuntimeClass();
ASSERT(AfxIsValidAddress((*pprc)->m_lpszClassName, sizeof(char*), FALSE));
// lame, but most classes will begin with 'C'
ASSERT((*pprc)->m_lpszClassName[0] == 'C');
}
__except (1)
{
returnfalse;
}
returntrue;
}

This procedure is risky, and is likely not 100% precise. My code employs structured exception handling in case some non-code memory is invoked by mistake. But the real danger is if the procedure misidentifies an actual virtual function that is not GetRuntimeClass(). Then, instead of calling GetRuntimeClass(), the algorithm may actually invoke some other function; perhaps the virtual destructor of the object it is examining. This would be bad, because the state of the object would likely change and would cause a severe error down the road. However, for the purposes of OpenVC, the procedure appears to work fine.

Finally, ShowInnards has two display modes. Report-mode uses Chris Maunder's grid control (with Ken Bertelson's grid-tree extensions) to display a report of the identified classes. For classes whose internal structure was identified, a right-side edit control shows a 'C'-style struct representation of the object's layout. Graph mode shows the entire class hierarchy in a CScrollView. I apologize for the sort of crappy tree-layout code, but I felt it was sufficient for the time being.

Knowing the internal layout of some of the classes in VC++ is key to implementing many of the features in WorkspaceEx. For example, WorkspaceEx tracks and updates a member variable of the CWorkspaceView class that stores the ordinal of the selected tab. Most addins will not have need to access or manipulate these structures, though some that attempt to modify the existing behavior of the IDE might. I could likely double the length of this article by discussing the finer points of what I've uncovered, but in truth it's not all that exciting. ShowInnards should provide enough information to get anyone started figuring out what some of the actual bits do. Beyond that, I recommend getting cozy with your debugger.

More Tricks & Tips

Hooking and Subclassing

The question I am asked most often about WorkspaceEx, is "how does it integrate so cleanly with the VC++ user interface?" The million-dollar answer (and I wish it would make me a million dollars) can be summed in four words: hooks and window subclassing.

WorkspaceEx, WndTabs, and other addins hook or subclass the windows created by Visual C++ so that the addin's code has a chance to handle window messages before the IDE does. These techniques are so powerful, that it is entirely possible to morph the application to look entirely different. If you don't believe me, check out RadVC. In combination with the ability to access and manipulate the internal MFC structures of the application, an Addin can do just about anything it wants.

Let me present an example of how this works. WorkspaceEx hooks a window of type CWorkspaceView. To find the correct window to hook, code inside WorkspaceEx enumerates the windows of the MSDEV process. The enumeration function calls a function similar to the following:

Note how the function determines the MFC class of the window in question. This makes for easy identification of the window. It may be possible to identify the window by other means (title, ID, window class, etc.), but this method is straightforward and works well.

The global variable g_wndWorkspace is of a type derived from CSubclassWnd, a class originally developed by Paul DiLascia. I recommend using this or a similar class for implementing window subclassing, since it makes simple work of the procedure.

In some scenarios it is beneficial to use an actual windows hook to intercept window messages. An example of how this is used in WorkspaceEx is with the Options dialog feature. WorkspaceEx inserts its own tab in the "tab control" of the options dialog in Visual C++. To do this, WorkspaceEx watches for the creation of the Options dialog using a WH_CBT hook, which is called whenever a new window is created in the process. In this case, the hook procedure looks for a window being created that matches some attributes of the Options dialog that I identified (namely name, size, and style). When a match is found, it subclasses that window using the same subclassing technique described above.

Toolbars

A shortcoming of the VC++ automation interface that I've heard several complaints about is the inability to specify a name for the addin's toolbar. Recall that VC++ allows addins to create a single toolbar with buttons that invoke the addin's commands.

This problem can be solved, to an extent, by using a windows CBT hook. Prior to calling AddCommandBarButton(), install the hook with a procedure that watches for the creation of windows with a title containing the string "Toolbar". The hook will be called as the toolbar window is created. Before the hook returns, update the window title to the name of your choice by modifying the window name found in the window creation structure passed to the hook. The caveat is that the new name must be the same or shorter length as the name it is replacing - usually eight characters. The hook procedure should look something like this:

There is a second problem with toolbar specification. VC++ documentation implies that the toolbar creation method, AddCommandBarButton(), should be called from inside the addin's OnConnection() method, but only when the OnConnection() parameter bFirstTime is set to VARIANT_TRUE. However, if your addin self-installs itself (as described earlier), OnConnection will never be called with this parameter set to VARIANT_TRUE. If you attempt to call the method from OnConnection() when bFirstTime is VARIANT_FALSE, the call will assert and fail.

This problem has a simpler solution. It turns out that the AddCommandBarButton() routine will not fail if it is called after the OnConnection() method returns, even if bFirstTime was never set to VARIANT_TRUE. You simply need to architect your addin to create its toolbar later than OnConnection(), and this can be done in many ways. I suggest creating a hidden window during OnConnection() (or hooking an existing window!) and posting a message to it before OnConnection() returns. When the message is processed, OnConnection() will have returned and the addin can call AddCommandBarButton(). If you use this technique, it is your own responsibility to know whether you've added your toolbar before; adding it twice or more would constitute bad manners.

Conclusion

That's about it. Putting these techniques to good use is left as an exercise for the reader. (Don't you just hate that?) On the off chance that I've riled up some Redmondtonian's with this expose, please consider offering me a job instead of a lawsuit. I don't have any assets you'd want anyway, other than my programming skill and dry wit.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

About the Author

Comments and Discussions

In the section Toolbars is some detail about how to change the name of a toolbar when the add-in is being added. I tried this procedure on MoreFast.Dll (another add-in listed in this area) without any success. Has anyone successfully done this?

When the user chooses an Add-in dll in the Tools Tab it calls first the DllRegisterServer () function from this Dll. After that it enumerates all Entries in the HKCR/CLSID hive and searches for a matching InprocServer32 Value.
This technique enables DevStudio to find all CoClasses the Add-in exports.
The first DllGetClassObject Call comes after closing the Customize Dialog.

I use 3 string table in my project : English / French / Neutral ...
If tou create a neutral table string with 20000 / 20001 / 20002 entries, when your add-in is loaded, if you create new project (dialog base for exemple) the Debug/release mode are replaced by you string entries 20000 ... 20002...
A lot of exemples are like this .. the resources of Add-In and Visual are shuffle...

Basically, do not let the IDE use your addin resources.
This happens in an MFC Extension DLL, if you leave the line:

new CDynLinkLibrary;

in your DllMain function.
Of course, if you remove that, you also won't be able to use your resources, unless you switch the resource instance handle before (and back after) you use them. See AfxGetResourceHandle and AfxSetResourceHandle.
Better if you wrap any externally callable function to a pair of calls to those functions (addin command methods and addin events, including OnConnection). I used a technique similar to AFX_MANAGE_STATE (a macro that instanciates an object performing the switch in its constructor/destructor).

Hope this helps,
Paolo

------Why spend 2 minutes doing it by hand when you can spend all night plus most of the following day writing a system to do it for you? - (Chris Maunder)

isn't it much simplier to get a pointer to the DllGetClassObject-export (through LoadLibrary and GetProcAddress) and instantiate suitable objects through its IClassFactory instead of hooking the registryaccess during DllRegisterServer?

I'm (for the joy of coding) trying to write an app in which to test AddIns. Ie a VisualStudio stub that loads the .dll and calls various methods, commands, events, etc.

Consequently I stumbeled into the stuff you describe - how does VS get the CLSID_DSAddIn for the add-in (i.e. how can the stub get it)? You mention listening to the registry and doing a DllRegisterServer. But RegNotifyChangeKeyValue isn't available in Win95 (as I vaguely recall it).

How to give you a new version of this Add-ins ?
I take your source code and add some new features :
-> Suppress bitmap and put Toolbar :
-> Easy way to add new button / new command (only ressource must be modified and command.h/cpp files)
-> Support multiple language (only English implemented, but can easilly support French / german, etc.)

=> Don't know how to make a project with 2 .def files (one un release, other for debug(finisehed by d) dll.

I want to give you this source code if you want to see...
I think it can be good added features...

Now who want to tell MFC is far too old? The problem is often how good can one design his software, instead of keep comparing MFC to WTL, putting blame on the framework. MSVC6 remains the most widely used C/C++ IDE even until today where VS.7.1 is in place.

Hi,
I have installed some software which made integration with VC6 via PKG file. After uninstalling the toolbar and other things are still present. I tried to remove PKG file which I suppose adds the toolbar but MSDEV does not startup saying: " filename.pkg This required file cannot be loadaed. Please re-install Microsoft Visual C++.". So the general problem is: how to prevent MSDEV to load the PKG file? Where are this information stored? Thanks for help!

(is this article talk all the MFC's Applications over?)
i am now doing some works with "Autodesk Inventor".
it is dependsed on the MFC42.dll. but the data in the AFX_MODULE_STATE is unavailable. i think it is because Inventor do not instantiation a CWinApp Object.

My Question:
1.how to work properly in a no-CWinApp-instance MFC Executable
2.can i retrieve the pointer to MFC Wnd Object by HWND