When calling this function (compiled for x64) the application just crashes. I can't even see any debug message in Visual Studio 2012. The identical code with the 32-bit-DLL (compiled for x86) works fine. The prototype is:

LPCSTR APIENTRY GetLastErrorText()

Unfortunately I don't have any further information about the DLL as it is a third-party product.

@AlexFarber As far as I know the default CharSet is Ansi. Anyway, the application still crashes. As mentioned, the 32 bit version is running fine.
–
RobertDec 18 '12 at 12:03

Why are not using AnyCPU and instead trying to target to specific platforms? The problem of course is the 64-bit process is likely trying to load the 32-bit version of the dll. My guess is that my64Bit.dll isn't actually compiled to be a 64-bit dll.
–
RamhoundDec 18 '12 at 12:35

1

@RobertStrauch - Sounds like you have lots of research to do.
–
RamhoundDec 18 '12 at 12:59

2 Answers
2

The function signature is quite troublesome. Whether your code will crash depends on what operating system you run. Nothing happens on XP, an AccessViolation exception is thrown on Vista and later.

At issue is that C functions returning strings need to typically do so by returning a pointer to a buffer that stores a string. That buffer needs to be allocated from the heap and the caller needs to release that buffer after using the string. The pinvoke marshaller implements that contract, it calls CoTaskMemFree() on the returned string pointer after converting it to a System.String.

That invariably turns out poorly, a C function almost never uses CoTaskMemAlloc() to allocate the buffer. The XP heap manager is very forgiving, it simply ignores bad pointers. Not the later Windows versions, they intentionally generate an exception. A strong enabler for the "Vista sucks" label btw, it took a while for programmers to get their pointer bugs fixed. If you have unmanaged debugging enabled then you'll get a diagnostic from the heap manager which warns that the pointer is invalid. Very nice feature but unmanaged debugging is invariably disabled when you debug managed code.

You can stop the pinvoke marshaller from trying to release the string by declaring the return value as IntPtr. You then have to marshal the string yourself with Marshal.PtrToStringAnsi() or one of its friends.

You still have the problem of having to release the string buffer. There is no way to do this reliably, you cannot call the proper deallocator. The only hope you have is that the C function actually returns a pointer to a string literal, one that's stored in the data segment and should not be released. That might work for a function that returns an error string, provided it doesn't implement anything fancy like localization. The const char* return type is encouraging.

You will need to test this to make sure there is no memory leak from not releasing the string buffer. Easy to do, call this function a billion times in a loop. If you don't get IntPtr.Zero as a return value and the program doesn't otherwise fall over with a out-of-memory exception then you're good. For 64-bit pinvoke you'll need to keep an eye on the test program's memory consumption.