Mistakes to avoid when designing DLLs

You've come here because you have either perpetrated one of the classic
design mistakes in your DLL, or asked a question similar to the
following:

What mistakes should I avoid making when designing my DLL ?

This is the Frequently Given Answer to that question.

Don't use compiler-specific function calling conventions.

Don't make the same numbskull mistake that some programmers at IBM once
made. They created a DLL in OS/2 Warp's TCP/IP subsystem
(tcpunx.dll) the functions in which used a function calling
convention that only IBM's own C/C++ compiler supported,
Optlink. (Since Optlink is the default with
IBM's C/C++ compiler if no calling convention is explicitly specified, it
is likely that IBM's programmers who wrote this particular DLL didn't
actually think about function calling conventions at all.) No-one using
any other C or C++ compiler, or indeed any other language, was thus able
to directly call any of the functions in tcpunx.dll.

For DLLs that may be called by code written in arbitrary languages and
compiled with arbitrary compilers, the best function calling convention to
use is the function calling convention employed by the operating system
API itself. This is the only function calling convention that is
guaranteed to have universal support by all of the implementations of all
of the languages (that support calling functions in DLLs, that is)
available on the platform.

For a 32-bit OS/2 DLL, this is the System calling convention.

For a Win32 DLL, this is the Stdcall calling convention.

For a Win16 DLL, this is the Pascal calling convention.

Don't mix heaps.

If you provide direct or indirect methods of allocating storage from your
language implementation's run-time heap, provide direct or indirect methods
of releasing that storage. Don't assume that the caller's
malloc() and free() functions (and
new and delete operators) operate upon the same
heap as the malloc() and free() functions (and
new and delete operators) that your library calls.
More often than not, they will not. Programs are heterogeneous things,
and may comprise many parts using different instances of the runtime
library. Indeed, they may comprise parts written in completely different
languages. You will end up deallocating with one (instance of a) compiler's
runtime library a piece of heap storage space that was allocated with a
completely different (instance of a) compiler's runtime library. This
is the non-stop express route to heap corruption.

So if (for example) you provide a function that allocates a buffer from heap
storage, populates it with data, and then returns it to the caller;
also provide a function that the caller can call to indicate that
it no longer wants that buffer. If you don't want to export individual
cleanup functions, simply export a wrapper around your runtime library's
free() function.

Similarly, if your library is the consumer of storage space
dynamically allocated by the caller, provide a function that is a wrapper
around your runtime library's malloc() function.

Alternatively, use the system-level heap management functionality provided
by the operating system itself. For 32-bit OS/2, allocate and release all
dynamically allocated memory that is shared between library and caller with
DosAllocMem() and DosFreeMem(). For Win32,
allocate and free all such dynamically allocated memory with
VirtualAlloc() and VirtualFree(). In both cases,
document this as part of your library's interface.

Don't play musical ordinal numbers.

If the import library for your DLL links to your entrypoints by ordinal,
ensure that the ordinals of the entrypoints in your library remain static
from release to release. This helps to preserve binary backwards
compatibility of new libraries with old applications.

This is less of a concern on Win32 than it is on other platforms, since,
strictly speaking, entrypoint ordinals are only hints on Win32.

Don't export internal details of your library.

Some compilers provide mechanisms to automatically export all functions and
variables in a library that have external linkage. Avoid using any such
mechanisms. Export exactly the interface that you need to export, and no
more. Exporting internal library details can:

… paint you into a corner.

If you export an internal detail of your library, then all too often, some
inquisitive programmer developing an important application will notice it
and use it. You will then be stuck with it as part of your interface.

… surprise users of your library.

If you export everything from your library, you might end up exporting a
symbol such as (say) fprintf. Depending from the order in which
import libraries are specified to the linker, calling applications may then
end up using your library's fprintf instead of the one in their
run-time library.