Stuff

PIX for Windows is a tool that comes with the Microsoft DirectX SDK for debugging Direct3D applications. I have a love/hate relationship with it, but when it comes to debugging rendering state problems, I haven't found anything that works better. NVPerfHUD is no good at tracking down single-frame glitches, and Intel Graphics Performance Analyzer (GPA), while surprisingly useful for debugging, doesn't expose full device state. Thus, I find myself using PIXWin a lot, even despite its shortcomings (its data presentation could use a lot of cleanup).

While trying to use it today, I ran into the annoyance of a PIX-only crash, meaning that the program would crash only when run with PIX instrumentation enabled.

Typically, I turn on the Direct3D debug runtime and make sure the program is debug runtime clean before trying to use PIX. The debug runtime fires debug asserts if it sees badness like invalid state, which is much easier to track down with the debug runtime than from mysterious crashes or error asserts in PIX. Unfortunately, that trick didn't work this time as the program ran cleanly outside of PIX and bombed inside of it, so I had to attach a debugger to the process while PIX had it hooked to try to diagnose the problem.

Long story short, the problem had to do with the extra code that the PIX hook runs when it instruments a process. PIX works by hooking the Direct3D entry points and running extra code to capture the D3D calls made by the program. Apparently, one of the things it does when D3D is initialized (Direct3DCreate9) is use Windows Management Instrumentation (WMI) to get some information about the graphics device. This in turn makes a blocking COM call to another thread, which then runs a message loop within the current thread to service messages while waiting for the other thread to run the call. This is needed to service other cross-thread calls back into the blocked thread and prevent deadlocks.

The other piece of the puzzle is a change that was made to COM starting with Windows Vista to allow WM_PAINT to be dispatched during blocking calls. IMO this is a bad idea as dispatching anything other than other sent messages (SendMessage) invites reentrancy problems, and that is true here as well. In my case, this caused a paint message to be dispatched on the display window that was still stuck in init in the Direct3DCreate9 call, resulting in an attempt to refresh the window using the 3D context that hadn't been inited yet. The workaround I applied was to add a reentrancy counter to block normal WM_PAINT handling on that window until init had completed.

4 comments | Aug 04, 2012 at 19:40 | default

Comments

Comments posted:

Thanks! I'm almost certain that the crash I have been getting in PIX is this exact one. For something like 6 months now I haven't been able to use PIX on our main game client. The issue would have started when added an initial loading screen to the client, which only renders frames when WM_PAINT messages were received.

The problem is that the time between doing that and the next point that I needed to use PIX was so long that there was no easy cause and effect chain.

I was still able to use PIX with some of our other tools that integrate our renderer, so I always recreated the problems there rather than trying to work out what the real problem was with our game client.

Looked at the call stack in the debugger. With the Microsoft Symbol Server and DirectX symbols hooked up, the debugger can unwind through both the OS window message dispatch and the PIX helper DLL, revealing the problem.

Phaeron - 05 08 12 - 11:14

I seem to recall at the time that when I attempted to attach the process to Visual Studio when run under PIX that it complained that there was already another debugger attached. Did you run in to anything like that? My memory of this may be wrong.

Nope, haven't run into that. In any case, there's a workaround for that too: attach WinDbg in non-invasive mode, and either debug with it directly or use it to generate a minidump that Visual Studio can run on.

Phaeron - 07 08 12 - 07:01

Comment form

Please keep comments on-topic for this entry. If you have unrelated comments about VirtualDub, the forum is a better place to post them.

Name:

Remember personal info?
YesNo

Email (Optional):

Your email address is only revealed to the blog owner and is not shown to the public.

URL (Optional):

Comment:

/

An authentication dialog may appear when you click Post Comment. Simply type in "post" as the user and "now" as the password. I have had to do this to stop automated comment spam.

Small print: All html tags except <b> and <i> will be removed from your comment. You can make links by just typing the url or mail-address.