There are many tutorials online for writing engines and shaders for shadow mapping. I have found an issue which seems to have a hard to find answer.

Typical shadow maps can be created by setting null render targets, assigning a depth/stencil, and setting a null pixel shader. What about the case where I want to sample a texture to determine an objects transparency and discard the pixel based on that transparency (i.e. shadow mapping a tree or chainlink fence)? In this case I need a pixel shader which samples a texture and outputs an alpha value to determine whether to write a pixel to the depth stencil. How do I output an alpha value if there are no render targets?

Where do I determine whether to write to the depth stencil based on the pixel shader operations? How do I output an alpha (0 or 1 in value)? Remember that the blend state is defined for each render target slot of which I have none.

\$\begingroup\$I do not quite understand why you want to output an alpha value? You should be able to sample the texture and then clip (-1) directly from the pixel shader that writes depth for any alpha sample that fails; no color output necessary. You will make the shadow map generation perform slightly worse by doing this (it violates certain hardware optimizations to arbitrarily discard a pixel during the evaluation of a pixel shader), but it might not be all that bad considering how simple the pixel shader is.\$\endgroup\$
– Andon M. ColemanMar 17 '15 at 20:17

\$\begingroup\$"I do not quite understand why you want to output an alpha value?" Think of generating the shadow map which contains a tree. A quad has a texture on it which is transparent between the leaves. I, thus, do not want to write to the depth stencil when the alpha value of the tree texture is 0. Is this a case where I would use discard() or clip(-1)?\$\endgroup\$
– Russell TrahanMar 17 '15 at 20:28

2

\$\begingroup\$Yes, that is absolutely where you would use discard (GLSL) or clip(-1) (HLSL). It tells the fragment/pixel shader to throw away the results for the current pixel; this includes color, depth, stencil. It is as though the pixel never existed. In fact, in modern OpenGL there is no fixed-function alpha test anymore, you actually have to implement alpha testing yourself using discard.\$\endgroup\$
– Andon M. ColemanMar 17 '15 at 20:59

\$\begingroup\$OK. Put that as answer and I'll mark it. I was hesitant to use clip/discard because I have been under the impression that having that statement in a shader prevents early-z testing.\$\endgroup\$
– Russell TrahanMar 17 '15 at 21:02

1

\$\begingroup\$It does prevent early Z, which I pointed out in parentheses in my original comment. The thing is, a fixed-function alpha test also disables early Z. Any test that can only be performed after or during execution of a pixel/fragment shader disables early Z, and alpha testing (whether you implement it yourself or use some fixed-function feature) can only be done after color is computed per-pixel.\$\endgroup\$
– Andon M. ColemanMar 17 '15 at 21:08

1 Answer
1

The proper way of doing this is to implement the alpha test yourself in a pixel shader and clip (-1) any pixel that fails. This will discard the pixel and no depth or stencil values will be written to the framebuffer, without having to write anything to a separate color buffer and perform an extra test afterwards.

As discussed in comments, arbitrarily deciding during pixel shader execution to discard a pixel is going to disable some hardware depth buffer optimizations. However, the complexity of your depth-only shader is such that it probably will not affect performance too much. Early Z benefits you most when you can skip complicated pixel shaders (such as those used in lighting) for occluded primitives. This shader basically amounts to a single (probably cached) texture lookup and comparison between 0.0 per-pixel; not complicated at all.