Saturday, January 11, 2014

VOGL OpenGL Tracer/Debugger - Bonus Content

There's a bunch of content I wanted to get into our Steam Dev Days presentation on our new OpenGL tracer/debugger (VOGL), but you can only cram in so much into a 20 minute presentation with 5-10 minutes set aside for demos. Here's the missing content:

20 comments:

Greetings Rich,Since you are very keen on Clang and QtCreator, do you have then work together? I mean there was some branch of QtCreator integrating Clang support ( code completion, etc.) but it is not merged. Thanks.

Duplicating questions here, just in case.What plans Valve have on this thing? Proprietary distribution like the AMD's gDEBugger?I'm working on open-source reincarnation of glslDevil: https://github.com/SirAnthony/GLSL-Debugger, actually integrating mesa gl compiller into it for GLSL 4.x support in shader debugging. May be you will be interested.﻿

We're going completely open source - GL is just too large of an API (and changing too quickly) for a small team to keep up. We'll be putting it up on bitbucket (or maybe github - we haven't decided for sure yet). We'll accept patches, bug reports, game traces, etc. We want to enable everyone that we can to make better GL games and drivers. We're planning on having the UI fleshed out enough to view all major GL state by GDC. Two major driver vendors have already expressed interest in helping us out which is wonderful.

vogl's tracer is just like apitrace's tracer: you can LD_PRELOAD it into your process (or manually load it like you would libgl.so) and it intercepts all of your GL/GLX calls. It's as transparent as we could make it: we shadow key bits, and optionally serialize what you do and pass along all calls into the driver. Occasionally, we have to make some extra hidden GL calls to retrieve linked program metadata, peek at GL errors, etc. or do glGet's while taking a snapshot. This activity should not be visible to your app. We are very careful about not injecting GL errors into your app while making hidden calls. We occasionally must modify your params during tracing, for example write-only mapped buffers are set as readable/writable when we call GL so we can read the exact contents of buffers when we serialize them.

If you use multiple contexts, we require that you don't leave your secondary contexts current on any threads when you call glXSwapBuffer()'s, because our snapshot code needs to make them current on the thread that calls glXSwapBuffer()'s in order to query their state. Also, we have some constraints on handle usage related to multiple contexts/different threads that we'll document. (These constraints should not impact the vast majority of GL apps in practice.) Shadowing GL handles, especially across multiple contexts in all possible (but technically valid) GL scenarios is very tricky.

We use two chroot's for cleanly building 32-bit and 64-bit. We don't use qmake, but cmake for building everything. It took us a while, but we figured out how to cleanly configure QtCreator so we can build the project via cmake from within chroots -- all nicely within QtCreator itself. We also figured out how to use QtCreator's ninja compiler warning/error parser to work when we use ninja to build (you need to redirect both stdout/stderr to just stdout, or maybe just stderr - I don't remember offhand how Mike did it exactly). Mike Sartain or I will write a full description of how we've deeply integrated cmake+ninja into QtCreator (just by configuring it properly within a couple days.

For clang, Mike Sartain figured out how to switch between gcc vs. clang in a way that was compatible with an "out of the box" install of QtCreator:http://linux-debugger-bits.blogspot.com/2013/07/clang-33-with-64-bit-ubuntu-1204.html

We first did this outside of chroots to build with clang. Now we do it within the 32-bit/64-bit chroot's. Works really well. Also, we've had no issues debugging clang built apps with QtCreator 2.8.x or 3.0.0.

>We're going completely open sourceSounds great. If you will do good interface for shader debugging entry points, I can port pieces of my code which implements your 'Really long term' and already working.

apitrace supports full-stream tracing only, which is fine for small GL apps but is less than optimal when trying to debug large GL games that can take 60-90 secs to load with additional time (dozens of secs up to several minutes) needed to get a repro case to show up. Playing back these large traces, sometimes dozens (even hundreds) of times, to narrow down bugs is not enjoyable or productive at all. There's a trim command in there but I gave up trying to use it after trying to trim a small TF2 trace and letting it go overnight (!).

Around a year ago Peter Lohrmann (working for Valve) actually spent several months modifying apitrace to do the things we need to effectively debug big GL apps: restorable state snapshots, single/multi-frame capturing while tracing or replaying, and efficient/usable trace trimming. His patches are out there, but I believe they were only tested on Source1 GL apps. This first attempt taught us a lot about the problem domain. Also, at the time apitrace was not able to reliably replay Source1 GL apps such as L4D2 without randomly diverging. Peter fixed these problems.

So after studying apitrace for quite a bit, and contributing back some fixes/patches, we realized the real problem we needed to solve (restorable state snapshotting) wasn't the focus of apitrace's architecture. So we didn't have a lot to loose by dumping it and starting from scratch. Also, apitrace's usage of Python scripts to auto-gen massive amounts of C/C++ code was a real turn-off, and we didn't care about D3D or ES debugging. In hindsight, I think we made the right decision because we've spent far more dev time on the state snapshot/restore problem than tracing.

About Me

Back in the day I worked for several years at Digital Illusions on things like the first shipping deferred shaded game ("Shrek" - 2001), software renderers, and game AI. Then, after working for Microsoft at Ensemble Studios for 5 years as engine lead on Halo Wars, I took a year off to create "crunch", an advanced DXTc texture compression library. I then worked 5 years at Valve, where I contributed to Portal 2, Dota 2, CS:GO, and the Linux versions of Valve's Source1 games. I was one of the original developers on the Steam Linux team, where I worked with a (somewhat enigmatic) multi-billionare on proving that OpenGL could still hold its own vs. Direct3D. I also started the vogl (Valve's OpenGL debugger) project from scratch, which I worked on for over a year. In my spare time I work on various open source lossless and texture compression projects: crunch, LZHAM, miniz, jpeg-compressor, and picojpeg.