The Stencil Test is a per-sample operation performed after the Fragment Shader. The fragment's stencil value is tested against the value in the current stencil buffer; if the test fails, the fragment is culled.

If the current framebuffer has no stencil buffer, then the stencil test will always behave as if it is disabled.

If there is a stencil buffer, that buffer has a certain bitdepth. This specifies the number of stencil bits available.

Fragment stencil value

Each Fragment has a stencil value, which is an unsigned integer. The stencil test operation will test this stencil value against the value from the current framebuffer at the fragment's position.

This value is usually defined by the same function that sets the stencil test, below. This means that every fragment from every primitive for every object in a draw call gets the same fragment stencil value (with the front/backface point noted below).

Stencil test

To enable stencil testing, call glEnable​ with GL_STENCIL_TEST. When rendering to a framebuffer that has no a stencil buffer, stenciling will always behave as if it is disabled.

Stenciling operations take into account the fact that triangles have two sides. Therefore, all stencil tests and operation functions have two sets of data: one for the front side of triangles and one for the back. Which side is used depends on whether the fragments generated from the primitive came from the front or back face.

These functions all take a face​ parameter, which specifies which facing state is set by that function. The face​ can be GL_FRONT or GL_BACK, but it can also set both sides at once with GL_FRONT_AND_BACK.

For Primitives that have no facing, the front side stencil state are always used.

The ref​ defines the fragment's stencil value for all fragments generated for the given facing. The fragment stencil value will be clamped to the range defined by the stencil buffer's bitdepth.

The first step of the stencil test is to get the destination stencil value from the stencil buffer (called Ds). This is an unsigned integer value. The fragment's stencil value will be called Fs.

The next step is to perform a bitwise AND with both Fs and Ds, against the mask​ parameter for both. This allows the user to mask off certain stencil bits, reserving them for different conditional tests. This results in two masked unsigned integers, Fm and Dm.

Then the stencil test itself is performed between Fm and Dm, based on the func​ parameter. The test is of the form (Fm FUNC Dm); the masked fragment value is on the left-hand side. The available functions are:

Enum

Test

Enum

Test

GL_NEVER

Always fails.

GL_ALWAYS

Always passes

GL_LESS

<

GL_LEQUAL

≤

GL_GREATER

>

GL_GEQUAL

≥

GL_EQUAL

=

GL_NOTEQUAL

≠

glStencilFunc​ can be used to set both face operations in one call, if you don't like to use GL_FRONT_AND_BACK.

Stencil operations

If the stencil test fails, the fragment is discarded. If the stencil test passes, other tests (like the Depth Test) may still discard the fragment. Normally, discarding a fragment means that it has no visible effects. However, the stencil buffer can be updated even from discarded fragments, if the depth or stencil tests discard them.

The parameters sfail​, dpfail​, and dppass​ define the stencil update operations to perform in the three cases above, respectively. Each case can use any of the following operations ("current value" here means the value already in the stencil buffer):

Enum

Operation

Enum

Operation

GL_KEEP

Don't modify the current value (default)

GL_INVERT

Invert the current value

GL_ZERO

Set it to zero

GL_REPLACE

Replace with the masked fragment value

GL_INCR

Increment the current value, saturating1 if it would overflow

GL_INCR_WRAP

Increment the current value, wrapping if it would overflow

GL_DECR

Decrement the current value, setting to zero if it would underflow

GL_DECR_WRAP

Decrement the current value, wrapping if it would underflow

1: Meaning that it stops at the maximum representable integer at the stencil buffer's bitdepth. For an 8-bit stencil buffer, that would be 255.

glStencilOp​ can be used to set both face operations in one call, if you don't like to use GL_FRONT_AND_BACK.