Tuesday, June 26, 2012

Who's fault is this?

Today I ran into an interesting phenomenon when writing a test application in Visual C++. The phenomenon appears to be a bug in Visual Studio 2008 optimizer, which seems to lose an update of a variable. Of course, it is equally possible that the bug is in my code, which does some really ugly stuff.

The idea of function test() is to return SUCCESS if and only if function f() triggers a guard page exception (and nothing else). The problem is that it does not work when using more aggressive optimizations.

Let's have a look at this on the assembly language level. Here are the relevant parts of function test(). At its beginning, we see register ESI loaded with the errstatus argument:

The compiler moves the value to the stat variable, which lives at address EBP - 0x1c on the stack. The next interesting place to look at is the piece of code which updates stat in the case of the guard page exception:

When the exception code, now stored in EAX is EXCEPTION_GUARD_PAGE, the stat variable on the stack is updated to 0x7D00 (SUCCESS). Because the __except expression evaluates to -1, execution of f() is resumed where it was previously interrupted. We'd expect f() to return to test() and test() to return the value now stored in stat. But looking at the end of function test() in the assembly, this is not how the compiler sees it:

return stat;011C12E1 mov eax,esi }

Instead of reading the stat variable from the stack, where the correct value is, the compiler returns the old value it previously loaded to ESI.

My impression is that either I am unintentionally doing something illegal, such as updating a local variable from the filter expression, or the compiler contains a bug which does not take the exception handling code into account when performing optimizations.

Well, one possible explanation which offers itself is that the compiler considers execution of the exception handling code somehow asynchronous to execution of test() and therefore requires stat to be defined volatile to make it work.