Tessellation is the stage in the OpenGL rendering pipeline where patches of vertex data are subdivided into smaller Primitives. This process is governed by two shader stages and a fixed-function stage.

Note: This describes the OpenGL 4.0 feature, not the old gluTess*​ tessellation functionality.

Overview

The tessellation process is divided into three stages which form an optional part of the rendering pipeline. Two of the stages are programmable; between them is a fixed function stage. They are described below, in the order they are processed.

Generally, the process of tessellation involves subdividing a patch of some type, then computing new vertex values (position, color, texture coordinates, etc) for each of the vertices generated by this process. Each stage of the tessellation pipeline performs part of this.

The Tessellation Control Shader (TCS) determines how much tessellation to do (it can also adjust the actual patch data, as well as feed additional patch data to later stages). Therefore, the TCS is primarily responsible for ensuring continuity across patches. So if you have two patches that need to have different levels of tessellation, the TCS invocations for the different patches need to use the proper tessellation controls to ensure that the interface between the patches uses the same amount of tessellation. Without this protection, gaps and breaks in what are supposed to be contiguous patches can occur.

The TCS is optional; default tessellation values can be used if no TCS is provided.

Patches

Tessellation stages operate on patches, a primitive type denoted by the constant GL_PATCHES. A patch primitive is a general-purpose primitive, where every n​ vertices is a new patch primitive. The number of vertices per patch can be defined on the application-level using:

with GL_PATCH_VERTICES as target and a value which has is on the half-open range [1, GL_MAX_PATCH_VERTICES]. The maximum number of patch vertices is implementation-dependent, but will never be less than 3.

Tessellation control shader

The first step of tessellation is the optional invocation of a tessellation control shader (TCS). The TCS has two jobs:

Determine the amount of tessellation that a primitive should have.

Perform any special transformations on the input patch data.

The TCS can change the size of a patch, adding more vertices per-patch or providing fewer. However, a TCS cannot discard a patch (directly; it can do so indirectly), nor can it write multiple patches. Therefore, for each patch provided by the application, one patch will be provided to the next tessellation stage.

The TCS is optional. If no TCS is active in the current program or program pipeline, then the patch data is passed directly from the Vertex Shader invocations to the tessellation primitive generation step. The amount of tessellation done in this case is taken from default values set into the context. These are defined by the following function:

When pname​ is GL_PATCH_DEFAULT_OUTER_LEVEL, values​ is a 4-element array of floats defining the four outer tessellation levels. When pname​ is GL_PATCH_DEFAULT_INNER_LEVEL, values​ is a 2-element array of floats defining the two inner tessellation levels.

These default values correspond to the TCS per-patch output variables gl_TessLevelOuter[4]​ and gl_TessLevelInner[2]​.

Tessellation primitive generation

Primitive generation is a fixed-function stage responsible for creating a set of new primitives from the input patch. This stage is only executed if a tessellation evaluation shader (TES) is active in the current program or program pipeline. Primitive generation is affected by the following factors:

The tessellation levels, provided either by the TCS or the default values, as stated above.

The spacing of the tessellated vertices, as defined by the subsequent TES stage. It may be equal_spacing​, fractional_even_spacing​, or fractional_odd_spacing​.

The input primitive type defined by the subsequent TES which may be one of triangles​, quads​ or isolines​. The TES can also force the generation of the tessellation as a series of points rather than triangles or lines by providing the point_mode​ primitive.

The primitive generation order defined by the subsequent TES, which may be cw​ or ccw​.

Abstract patch

Notice that the primitive generation is not affected by the outputs of the TCS, the TCS's output patch size, any per-patch TCS outputs (besides the tessellation levels), and so forth. The primitive generation part of the tessellation stage is completely blind to the actual vertex coordinates and other patch data.

The purpose of the primitive generation system is to determine how many vertices to generate, in which order to generate them, and what kind of primitives to build out of them. The actual per-vertex data for these vertices, such as position, color, etc, is to be generated by the TES, based on information provided by the primitive generator.

Because of this dichotomy, the primitive generator operates on what could be considered an "abstract patch". It doesn't look at the patch output from the TCS; it thinks only in terms of tessellating an abstract quad, triangle, or "isoline" block.

Depending on the abstract patch type, the primitive generator evaluates a different number of tessellation levels and applies different tessellation algorithms. Each generated vertex has a normalized position (i.e. the coordinates are on the range [0, 1]) within the abstract patch. This position has two or three components, depending on the type of the patch. The coordinates are provided to the TES via the built-in in vec3 gl_TessCoord​ input.

Tessellation levels

The amount of tessellation that is done over the abstract patch type is defined by inner and outer tessellation levels. These, as previously stated, are provided either by the TCS or by context parameters specified via glPatchParameter​. They are a 4-vector of floats defining the "outer tessellation levels" and a 2-vector of floats defining the "inner tessellation levels."

The specific interpretation depends on the abstract patch type being used, but the general idea is this. In most cases, each tessellation level defines how many segments an edge is tessellated into; so a tessellation level of 4 means that an edge will become 4 edges (2 vertices become 5). The "outer" tessellation levels define the tessellation for the outer edges of the primitive. This makes it possible for two or more patches to properly connect, while still having different tessellation levels within the patch. The inner tessellation levels are for the number of tessellations within the abstract patch.

Not all abstract patches use the same number of values in the outer/inner tessellation levels data. For example, triangles only uses one inner level and 3 outer levels. The rest are ignored.

The tessellation levels specified in this way are not directly used. They go through a clamping process to generate the effective tessellation levels that are used to tessellate the primitive. This process depends on the TES's spacing parameter.

In the below discussion, max​ is the maximum allowed tessellation level, as defined by the GL_MAX_TESS_GEN_LEVEL. It must be at least 64, so you have some room to play with.

The spacing affects the effective tessellation level as follows:

equal_spacing​

Each tessellation level is individually clamped to the closed range [1, max​]. Then it is rounded up to the nearest integer to give the effective tessellation level.

fractional_even_spacing​

Each tessellation level is individually clamped to the closed range [2, max​]. Then it is rounded up to the nearest even integer to give the effective tessellation level.

fractional_odd_spacing​

Each tessellation level is individually clamped to the closed range [1, max​ - 1]. Then it is rounded up to the nearest odd integer to give the effective tessellation level.

The patch can be discarded if any outer tessellation level that is used by the abstract patch type is 0 or less. It can also be discarded if it is a floating-point NaN. A patch that is discarded does not get tessellated, and no TES is invoked for it. It is simply swallowed by the system as though it never were.

This allows a TCS to effectively cull patches by passing 0 for a relevant outer tessellation level.

Edge tessellation spacing

At various points in the forthcoming discussion about tessellating the abstract patch, there will be statements that say to tessellate an edge of some primitive. This means to subdivide it into a series of segments according to some tessellation level. The total length of these segments is the length of the original segment. However, the length of individual segments depends on the spacing parameter specified in the TES.

Given an effective tessellation level, denoted by n​, which applies to that edge, the vertices for an edge tessellated by n​ is defined as:

equal_spacing​

The edge is divided into n​ segments. All segments will have equal length.

fractional_even_spacing​, fractional_odd_spacing​

If n​ is 1, then no subdivision occurs. Otherwise, the edge will be divided into n​ - 2 segments of equal length. There will also be 2 segments that have length equal to each other, but not necessarily to the first group. The length of these 2 segments, relative to the others, will be n​ - f​, where f​ is the effective tessellation level value after clamping but before being rounded up.

When n​ == f​, the length of the 2 segments will be equal to the length of the other segments. As n​ - f​ approaches 2.0, the relative length of the 2 segments approaches 0.0.

The exact location of the 2 shorter segments is not defined, but they should be placed symmetrically, on opposite sides of the subdivided edge. Also, the location must be invariant with the same f​ value (thus allowing tessellated edges to work together).

The purpose of the fractional spacing modes is to have smoother, more stable interpolation as tessellation levels change. This is best used if tessellation levels are based on the distance to the camera or something.

Triangles

The abstract patch of the triangle tessellation is a triangle, naturally. Only the first three outer tessellation levels are used, and only the first inner tessellation level is used.

Each vertex generated and sent to the TES will be provided Barycentric coordinates as the gl_TessCoord​ input. This defines where this vertex is located within the abstract triangle. With this coordinate, it is possible to multiply any vertex attribute from 3 vertices to compute the appropriate value from the tessellation unit.

Quads

The abstract patch of a quad is a rectangle, naturally. All 4 outer and 2 inner tessellation levels are used.

Each vertex generated and sent to the TES will be provided a normalized 2D coordinate as the gl_TessCoord​ input, representing the location of that vertex within the abstract patch.

Isolines

The abstract patch of isolines is a rectangle, oddly enough. Only the first two outer tessellation levels are used; none of the inner tessellation levels are used.

The rectangular abstract isolines patch represents a series of horizontal lines. The first outer tessellation level defines how many segments the lines are divided into, and the second outer level defines how many lines are in the patch. So if you want to tessellate a single line, you should pass 1 for gl_TessLevelOuter[1]​. However, the TES is perfectly capable of stitching multiple separate lines together, even though they are parallel in the abstract patch.

Each vertex generated and sent to the TES will be provided a normalized 2D coordinate as the gl_TessCoord​ input. The y​ component specifies which line (normalized to the half-open range [0, 1) ) is being generated. The x​ component defines how far along that line this vertex should be generated for.

Tessellation evaluation shader

The Tessellation Evaluation Shader (TES) is responsible for taking the abstract coordinates generated by the primitive generator, along with the outputs from the TCS (or vertex shader, if no TCS is used), and using them to compute the actual values for the vertices. This is where you code the algorithm that you actually use to compute the new positions/normal/texcoords/etc. The TES is a mandatory part of tessellation; if one is not present, then tessellation doesn't happen.

The TES is rather like a vertex shader, in that each invocation operates on a distinct vertex within the tessellated patch (though, as with a vertex shader, the system may call the TES more than once for the same vertex, so it should be deterministic). Also, the TES cannot cull vertices.