Stuff

I ran into an annoying problem yesterday while rewriting some of VirtualDub's assembly language routines for AMD64.

Take this innocuous looking routine, for instance:

.code
trap proc public
sub rsp, 256
int 3
add rsp, 256
trap endp
end

It just fires the application crash handler with a breakpoint exception. Well, actually, not quite. What it actually does is terminate the application. The reason has to do with the way exception handling works on AMD64 Windows.

In Win32, active exception handlers are kept together in a
singly-linked list, pointed to by the Thread Environment Block (TEB).
The TEB, is in turn, set as the base of the FS selector, so that FS:[0]
is the head of the exception handler list. When an exception occurs,
such as from a C++ throw statement, the OS walks the list,
looking for a handler that understands the exception. When it finds
one, it then executes the handler, and optionally executes the handlers
on the list down to that point again to unwind the stack. If no handler
accepts the exception, the OS invokes the unhandled exception filter
set by the application, or UnhandledExceptionFilter() if none is set.
VirtualDub plugs in its own handler, which unwinds the app's context
stack to show the "friendly" dialog, then dumps the disassembly and
stack to create the crashinfo.txt file.

I also abuse the FS:[0] SEH chain to hold the stack pointer when I
need to reuse ESP as an eighth register, but that's a different story.

AMD64 dumps this method of EH and switches to table-based
exception handling. Instead of manually setting up an exception
handling chain in code, instead the compiler emits tables indicating
how program locations correspond to functions, which exception handlers
to use, and how to unwind. The benefit is that no explicit
linking/unlinking is required and thus the runtime overhead of
try/catch blocks is minimal (there is still some overhead in making
sure there are clear exception handling points in the code). The
disadvantages are that the exception handling tables can be rather
large, and that they must exist for every function that may need
to be unwound when an exception is fired.

Let me repeat that again: the tables must exist for every function that may need to be unwound.
Not just functions that throw, or functions that have explicit or
implicit try/catch blocks. Every function that has a stack frame needs
an EH structure; the exception is functions that do not allocate stack
space and thus have no stack frame (leaf functions). The assembly
function above has a stack frame but has no exception handling tables
declared. So the OS attempts to unwind past the function, fails, and
then simply terminates the application. No exception handler fires, no
crash dump is produced, and the program simply vanishes.

This, frankly, is unacceptable. Well, one way to fix it is to insert directives so the necessary tables are created:

This allows the OS to unwind the function, and exception handling
works. Thankfully, you only need to annotate instructions that alter
the stack layout, not every instruction. What I don't like is how easy
it appears to accidentally sabotage the exception handling system so
that any subsequent exception is uncatchable. All it takes for one
driver to load and run asm code without the correct tables -- if it
crashes anywhere in that code, the application will simply die, with no
hope for a crash report to figure out what happened. Given the number
of problems I've seen in the past with third-party codecs and
libraries, this scares me.

What I need is a hook that gets executed when the stack unwind
fails, but so far I haven't found one yet. There is a facility to plug
in a function to produce tables for dynamically generated code, which
is a possibility; in theory I could write a hook that creates fake RUNTIME_FUNCTION
structures for the code that crashes and point the OS to my crash
handler. The huge problem with this is distinguishing a valid crash in
a leaf function from a crash that won't unwind. A third possibility is
to figure out where the OS code is that terminates the app on an unwind
failure and patch it, but I'm sure the Windows guys just love dealing with that kind of code.

5 comments | Sep 01, 2004 at 03:34 | default

Comments

Comments posted:

I've been using Virtual Dub since the end of the 1.4.x releases. Thanks for all your hard work and for making it available! (Also, I was going to post this under the 1.6.0 topic, but the comment form wasn't loading for it.)

Thanks for providing VirtualDub - its cooool. I have one another question - I need some software for creating VCD which is freely available on the net and is good. Do you know of any software which I can use to convert my mpegs and avi files to VCD? Thanks.

fiza - 08 09 04 - 16:59

Hi friends,

Once more i thank you for remembering about poor peple like me who wants to edit video files. I have one small video i edited with virtual dub. If you want i can send it to you to see.

My wish with virtual dub is to insert parts of videos and insert blank spaces.