GPU shadow volume extrusion with GLSL

So, I'm about to transition from CPU to GPU shadow volume extrusion. This being said, I understand the CPU approach ( I wrote it! ) but being a novice shader programmer, I hope people can help me a little with the math, since shaders live in a slightly different space.

First, I believe I understand the principle for GPU shadow volume extrusion. I have three VBOs -- one for the triangle geometry which makes up the light facing caps and the extruded caps. One for per-vertex normals so I know I'm getting the triangle's normal for each vertex ( I know this means vertex duplication ). And finally, one VBO which contains doubled-up edges as quads for silhouette extrusion. My understanding is that I can use the same shader for silhouette extrusion as for volume caps.

Now, in my CPU implementation the approach I take is this:

1) I use the inverse of the modelview matrix for the model to transform the light position to the untransformed space of the model.

2) I do the extrusion to infinity more or less like so ( leaving out silhouette math. Also, the code below is for positional light, not directional )

It looks like it will work, but I think you're thinking a bit too much about the extrusion direction. What I did was transform the normal as you normally would, then do the dot product with the light as you did. If it's negative, I would extrude not the point, but the transformation matrix itself. In this case, I took took the light direction scaled by my extrusion amount, and subtracted the modelview[0].w, modelview[1].w, and modelview[2].w by the extrusion amount's x, y, and z respectively. I then transformed the position as I normally would. This way I get around the orientation by feeding the transformation directly into the matrix.

I know that you've decided to stick with stencil shadows for now, but I've just came across a new shadow mapping method: trapezoidal shadow mapping. It seems to take only a little more work to calculate the matrix for the light, and a shader to correctly offset the depth values, (a good approximation can be done solely with vertex shaders) while it gives probably the best shadow quality presently available on current graphics cards. You should take a look at it, since I'm sure you'll eventually go to shadow maps. http://www.comp.nus.edu.sg/~tants/tsm/TSM_recipe.html

That's really quite interesting. The whole reason I've avoided shadow maps is due to the trickery required to get decent precision. That being said, while I will read the article a few times and try to understand, I fear the math's over my head.

Actually, it wasn't a winding issue at all -- I was passing the wrong normal array when rendering the edge quads. Whoops.

That being said, my extrusion shaders work ( both directional and positional ) -- YAY! Sort of. I'm getting weird rendering side effects. I'm working on it now, but I'm likely to post a plea for help with screenshots and example code soon...

Now, in the demo I used with three light sources I get 60fps, instead of 30, with very little CPU hit outside of visibility determination.

In my stress test, however, I get the same FPS as with the CPU implementation ( about 20fps). But that's great, since my CPU is barely being touched, and thus has more cycles to dedicate to physics and AI. Also my stress test really is a "stress test", and unrepresentative of my intended usage.