Shader Compilation

Shader Compilation is the process of text in the OpenGL Shading Language and loading it into OpenGL to be used as a Shader. OpenGL has three ways to compile shader text into useable OpenGL objects. All of these forms of compilation produce a Program Object.

Note: This article will contain references to OpenGL 4.x features, such as tessellation shaders or Compute Shaders. If you are using OpenGL 3.x, you can ignore such references.

Shader and program objects

A Program Object can contain the executable code for all of the Shader stages, such that all that is needed to render is to bind one program object. Building programs that contain multiple shader stages requires a two-stage compilation process.

This two-stage compilation process mirrors the standard compile/link setup for C and C++ source code. C/C++ text is first fed through a compiler, thus producing an object file. To get the executable code, one or more object files must be linked together.

With this method of program creation, shader text is first fed through a compiler, thus producing a shader object. To get the executable program object, one or more shader objects must be linked together.

Shader object compilation

The first step is to create shader objects for each shader that you intend to use and compile them. To create a shader object, you call this function:

This creates an empty shader object for the shader stage given by given shaderType​. The shader type must be one of GL_VERTEX_SHADER, GL_TESS_CONTROL_SHADER, GL_TESS_EVALUATION_SHADER, GL_GEOMETRY_SHADER, GL_FRAGMENT_SHADER, or GL_COMPUTE_SHADER. Note that the control and evaluation shaders require GL 4.0 (or ARB_tessellation_shader), and the compute shader requires GL 4.3 (or ARB_compute_shader).

Once you have a shader object, you will need to give it the actual text string representing the GLSL source code. That is done via this function:

This function takes the array of strings, given by string​ and stores it into shader​. Any previously stored strings are removed. count​ is the number of individual strings. OpenGL will copy these strings into internal memory.

When the shader is compiled, it will be compiled as if all of the given strings were concatenated end-to-end. This makes it easy for the user to load most of a shader from a file, but to have a standardized preamble that is prepended to some group of shaders.

The length​ can be either NULL or an array of count​ integers. These are the lengths of the corresponding strings in the string​ array. This allows you to use non-NULL-terminated strings. If you pass NULL, then OpenGL will assume all of the strings are NULL-terminated and will therefore compute the length in the usual way.

Once shader strings have been set into a shader object, it can be compiled with this function:

Shader error handling

Compilation may or may not succeed. Shader compilation failure is not an OpenGL Error; you need to check for it specifically. This is done with a particular call to glGetShaderiv​:

GLintsuccess=0;glGetShaderiv(shader,GL_COMPILE_STATUS,&success);

If success​ is GL_FALSE, then the most recent compilation failed. Otherwise, it succeeded.

Shader compilation is pass/fail, but it is often useful to know why. This, like in most languages, is provided as text messages. OpenGL allows you to query a log containing this information. First, you must use glGetShaderiv​ to query the log's length:

GLintlogSize=0;glGetShaderiv(shader,GL_INFO_LOG_LENGTH​,&logSize);

This tells you how many bytes to allocate; the length includes the NULL terminator. Once you have the length and have allocated sufficient memory, you can use this function to get the log:

maxLength​ is the size of infoLog​; this tells OpenGL how many bytes at maximum it will write into infoLog​. length​ is a return value, specifying how many bytes it actually wrote into infoLog​; you may pass NULL if you don't care.

Shader compilation error checking.

GLuintshader=glCreateShader(...);// Get strings for glShaderSource.glShaderSource(shader,...);glCompileShader(shader);GLintisCompiled=0;glGetShaderiv(shader,GL_COMPILE_STATUS,&isCompiled);if(isCompiled==GL_FALSE){GLintmaxLength=0;glGetShaderiv(shader,GL_INFO_LOG_LENGTH,&maxLength);// The maxLength includes the NULL characterstd::vector<GLchar>errorLog(maxLength);glGetShaderInfoLog(shader,maxLength,&maxLength,&errorLog[0]);// Provide the infolog in whatever manor you deem best.// Exit with failure.glDeleteShader(shader);// Don't leak the shader.return;}// Shader compilation is successful.

Program setup

One you have successfully compiled the shader objects of interest, you can link them into a program. This begins by creating a program object via this command:

Note: It is possible to attach multiple shader objects for the same shader stage to a program. This is perfectly legal. When linking them, all of the code will be combined. However, only one of those shader objects can have a main​ function. Linking like this will work very much like linking multiple object files in a C/C++ program: one .cpp file can call code in another .cpp file if it has forward declarations of functions. So you can have "libraries" of functions that individual shaders can choose to use.

Recommendation: That being said, while this power is available, it is best not to use it. It usually works, but because most OpenGL applications don't do this, it doesn't get as thoroughly tested as other parts of the OpenGL API. So you're likely to run into more driver bugs this way. Generally stick to having one shader object per shader stage.

Before linking

A number of parameters can be set up that will affect the linking process. This generally involves interfaces with the program. These include:

Linking and variables

Normally, shader objects for different shader stages don't interact. Each shader stage's code is separate from others. They have their own global variables, their own functions, etc.

This is not the case entirely. Certain definitions are considered shared between shader stages. Specifically, these include uniforms, buffer variables, and buffer-backed interface blocks.

If one of these is defined in one stage, another stage can define the same object with the same name and the exact same definition. If this happens, then there will only be one uniform/buffer variable/interface block visible from the introspection API. So shader stages in the same program can share uniform variables, allowing the same value to be set into both stages with one glUniform​ call.

For this to work however, the definitions must be exactly the same. This includes the order of the members, any user-defined data structures they use, array counts, everything. If two definitions in different stages have the same name, but different definitions, then there will be a linker error.

Cleanup

After linking (whether successfully or not), it is a good idea to detach all shader objects from the program. This is done via this function:

If you do not intend to use this particular shader object in the linking of another program, you may delete it. This is done via glDeleteShader​. Note that the deletion of a shader is deferred until the shader object is no longer attached to a program. Therefore, it is a good idea to detach shaders after linking.

Example

//Read our shaders into the appropriate buffersstd::stringvertexSource=//Get source code for vertex shader.std::stringfragmentSource=//Get source code for fragment shader.//Create an empty vertex shader handleGLuintvertexShader=glCreateShader(GL_VERTEX_SHADER);//Send the vertex shader source code to GL//Note that std::string's .c_str is NULL character terminated.constGLchar*source=(constGLchar*)vertexSource.c_str();glShaderSource(vertexShader,1,&source,0);//Compile the vertex shaderglCompileShader(vertexShader);GLintisCompiled=0;glGetShaderiv(vertexShader,GL_COMPILE_STATUS,&isCompiled);if(isCompiled==GL_FALSE){GLintmaxLength=0;glGetShaderiv(vertexShader,GL_INFO_LOG_LENGTH,&maxLength);//The maxLength includes the NULL characterstd::vector<GLchar>infoLog(maxLength);glGetShaderInfoLog(vertexShader,maxLength,&maxLength,&infoLog[0]);//We don't need the shader anymore.glDeleteShader(vertexShader);//Use the infoLog as you see fit.//In this simple program, we'll just leavereturn;}//Create an empty fragment shader handleGLuintfragmentShader=glCreateShader(GL_FRAGMENT_SHADER);//Send the fragment shader source code to GL//Note that std::string's .c_str is NULL character terminated.source=(constGLchar*)fragmentSource.c_str();glShaderSource(fragmentShader,1,&source,0);//Compile the fragment shaderglCompileShader(fragmentShader);glGetShaderiv(fragmentShader,GL_COMPILE_STATUS,&isCompiled);if(isCompiled==GL_FALSE){GLintmaxLength=0;glGetShaderiv(fragmentShader,GL_INFO_LOG_LENGTH,&maxLength);//The maxLength includes the NULL characterstd::vector<GLchar>infoLog(maxLength);glGetShaderInfoLog(fragmentShader,maxLength,&maxLength,&infoLog[0]);//We don't need the shader anymore.glDeleteShader(fragmentShader);//Either of them. Don't leak shaders.glDeleteShader(vertexShader);//Use the infoLog as you see fit.//In this simple program, we'll just leavereturn;}//Vertex and fragment shaders are successfully compiled.//Now time to link them together into a program.//Get a program object.GLuintprogram=glCreateProgram();//Attach our shaders to our programglAttachShader(program,vertexShader);glAttachShader(program,fragmentShader);//Link our programglLinkProgram(program);//Note the different functions here: glGetProgram* instead of glGetShader*.GLintisLinked=0;glGetProgramiv(program,GL_LINK_STATUS,(int*)&isLinked);if(isLinked==GL_FALSE){GLintmaxLength=0;glGetProgramiv(program,GL_INFO_LOG_LENGTH,&maxLength);//The maxLength includes the NULL characterstd::vector<GLchar>infoLog(maxLength);glGetProgramInfoLog(program,maxLength,&maxLength,&infoLog[0]);//We don't need the program anymore.glDeleteProgram(program);//Don't leak shaders either.glDeleteShader(vertexShader);glDeleteShader(fragmentShader);//Use the infoLog as you see fit.//In this simple program, we'll just leavereturn;}//Always detach shaders after a successful link.glDetachShader(program,vertexShader);glDetachShader(program,fragmentShader);

Separate programs

A program object can contain the code for multiple shader stages. The glUseProgram​ function only takes a single program, so you can only use a single program at a time for rendering. Therefore, you cannot mix-and-match code for different shader stages dynamically post-linking. Shader objects are not programs; they only hold compiled fragments of code, not fully useful programs.

There is a way to do this. This involves two alterations to the model presented above. The first is how the program is created; the second is in how it gets used.

To allow the use of multiple programs, were each program only provides some of the shader stage code, we must first create our programs specially. To signal that a program object is intended to be used with this separate program model, we must set a parameter on the program before linking. This is done as follows:

glProgramParameter(program,GL_PROGRAM_SEPARABLE,GL_TRUE);

Note: Again, this must done be before the program is linked.

There is an alternative method for creating separable programs. This represents a common use case: creating a program from a single set of shader source which provides the code for a single shader stage. The function to do this is:

This works exactly as if you took count​ and strings​ strings, created a shader object from them of the type​ shader type, and then linked that shader object into a program with the GL_PROGRAM_SEPARABLE parameter. And then detaching and deleting the shader object.

This process can fail, just as compilation or linking can fail. The program infolog can thus contain compile errors as well as linking errors.

Note:glCreateShaderProgramv​ will return either the name of a program object or zero - independent of which errors might occur during shader compilation or linkage! A return value of zero simply states that either the shader object or program object could be created. If a non-zero value is returned, you will still need to check the program info logs to make sure compilation and linkage succeeded! Also, the function itself my generate an error under certain conditions which will also result in zero being returned.

Warning: When linking shaders with separable programs, your shaders must redeclare the gl_PerVertex interface block if you attempt to use any of the variables defined within it.

Separable programs are allowed to have shaders from more than one stage linked into them. While it is best to only use shaders from one stage (since the main point of using separable programs is the ability to mix-and-match freely), you do not have to. However, if two stages are linked together in the same program, you will be unable to insert another program between those two stages, due to pipeline validation rules.

Program pipelines

Creating a separable program is just the first step. The other thing you must do is change how the program is used.

To use multiple separable programs, they must first be assembled into an OpenGL Object type called a program pipeline. Unlike program or shader objects, these follow the standard OpenGL Object mode. Therefore, there is a glGenProgramPipelines​ function to create new pipeline names, a glDeleteProgramPipelines​ to delete them, and a glBindProgramPipeline​ to bind it to the context. Program pipeline objects do not have targets, so the last function only takes the pipeline to be bound.

Similar to Sampler Objects, program pipeline objects should only be bound when you intend to render with them (or set uniforms through them, as described below). The only state in program pipeline objects are the list of programs that contain the code for the various shader stages. This state is set by this function:

The given pipeline​ will get the shader code for the shader stages defined by the bitfield stages​ from the given program​. The stages​ bitfield determines which shader stages in program​ will provide code for those shader stages in the pipeline. These bits can be a combination of GL_VERTEX_SHADER_BIT, GL_TESS_CONTROL_SHADER_BIT, GL_TESS_EVALUATION_SHADER_BIT, GL_GEOMETRY_SHADER_BIT, GL_FRAGMENT_SHADER_BIT and GL_COMPUTE_SHADER_BIT. The bitfield can also be GL_ALL_SHADER_BITS, which is equivalent to all of the above. If the program​ has an active code for each stage mentioned in stages​, then that code will be used by the pipeline. If program​ is 0, then the given stages are cleared from the pipeline.

Rendering

Once you have a functioning program pipeline with all of the separate stages you would like to use, you can render with it. To do that, you must first bind the program pipeline with glBindProgramPipeline​.

Warning:glUseProgram​overridesglBindProgramPipeline​. That is, if you have a program in use and a program pipeline bound, all rendering will use the program that is in use, not the pipeline programs. So make sure that glUseProgram(0)​ has been called.

Uniforms and pipelines

For each uniform call, the extension introduced a corresponding direct state access (DSA) pendant. This family of new functions is summarized as glProgramUniform​. Since these functions access the state of the corresponding program object directly, first activating the program is not necessary, i.e. a call to glUseProgram​ is obsolete for the purpose of modifying uniform values. glProgramUniform​​ works on any program object, separable or not. Separability has no influence on location queries. glGetUniformLocation​ will work with any program object that has been successfully linked. Using the DSA-style functions is particularly efficient with core OpenGL 4.3 or if GL_ARB_explicit_uniform_location is present. In that case, neither location queries have to be done nor is making a program object current or active necessary.

Recommendation: The following explains a highly convoluted way to use glBindProgramPipeline​ to allow the user to set uniform state for one of the programs within a pipeline. You are strongly advised to ignore this section. Instead, when writing new code, use glProgramUniform​ to set uniforms directly into programs without having to bind or use the program. Only use this if you absolutely must remain backwards compatible with some functions that set uniforms on a program that is currently in use.

Trying to use a call from the glUniform​ family of functions will not work with program pipelines unless certain conditions are met. First, the program pipeline the program is bound to must be current. Second, glActiveShaderProgram​ must first be called to select the active program object in the pipeline. Using glActiveShaderProgram​ will not actually make the program current, but simply functions as a selector. Uniform updates will then be redirected to the active program object. A separable program can also be made current by calling glUseProgram​. Uniform update will then work as usual.

Examples of separate programs

The following examples depict multiple possible scenarios when using separable programs. The first example aims at showing the simplicity inherent in using glCreateShaderProgramv​. The second shows how to deal with multi-stage programs and doing some pre-linking work not possible when creating single-stage, separable programs.

Note: For simplicity, the following code assumes that source strings result in successful compilation and linking and all resources involved are created successfully. However, it is a very good idea to do some error checking in real code and get the shader/program/pipeline logs! As a minimum, you should do as shown in the previous example for monolithic prorams.

Two separate programs for vertex and fragment shading

Creating two separable programs, one with a vertex shader and one with a fragment shader.

//Create two separable program objects from a //single source string respectively (vertSrc and fragSrc)GLuintvertProg=glCreateShaderProgramv(GL_VERTEX_SHADER,1,&vertSrc);GLuintfragProg=glCreateShaderProgramv(GL_FRAGMENT_SHADER,1,&fragSrc);//CHECK FOR ERRORS HERE!.//Generate a program pipeline and attach the programs to their respective stagesglGenProgramPipelines(1,&pipeline);glUseProgramStages(pipeline,GL_VERTEX_SHADER_BIT,vertProg);glUseProgramStages(pipeline,GL_FRAGMENT_SHADER_BIT,fragProg);//Query and set any uniformsGLintcolorLoc=glGetUniformLocation(fragProg,"Color");glProgramUniform4f(fragProg,colorLoc,1.f,0.f,0.f,1.f);

Mixing a single- and multi-stage programs

Creates a separate program, where some of the stages are directly linked together.

//Create two programs. One with just the vertex shader, and //one with both geometry and fragment stages.GLuintvertexProgram=glCreateProgram();GLuintgeomFragProgram=glCreateProgram();//Declare that programs are separable - this is crucial!glProgramParameteri(vertexProgram,GL_PROGRAM_SEPARABLE,GL_TRUE);glProgramParameteri(geomFragProgram,GL_PROGRAM_SEPARABLE,GL_TRUE);//Generate and compile shader objects, as normal.GLuintvertShader=glCreateShader(GL_VERTEX_SHADER);GLuintgeomShader=glCreateShader(GL_GEOMETRY_SHADER);GLuintfragShader=glCreateShader(GL_FRAGMENT_SHADER);glShaderSource(vertShader,1,&vertSrc,NULL);glShaderSource(geomShader,1,&geomSrc,NULL);glShaderSource(fragShader,1,&fragSrc,NULL);glCompileShader(vertShader);glCompileShader(geomShader);glCompileShader(fragShader);//Attach the shaders to their respective programsglAttachShader(vertexProgram,vertShader);glAttachShader(geomFragProgram,geomShader);glAttachShader(geomFragProgram,fragShader);//Perform any pre-linking steps.glBindAttribLocation(vertexProgram,0,"Position");glBindFragDataLocation(geomFragProgram,0,"FragColor");//Link the programsglLinkProgram(vertexProgram);glLinkProgram(geomFragProgram);//Detach and delete the shader objectsglDetachShader(vertexProgram,vertShader);glDeleteShader(vertShader);glDetachShader(geomFragProgram,geomShader);glDetachShader(geomFragProgram,fragShader);glDeleteShader(geomShader);glDeleteShader(fragShader);//Generate a program pipelineglGenProgramPipelines(1,&pipeline);//Attach the first program to the vertex stage, and the second program//to the geometry and fragment stagesglUseProgramStages(pipeline,GL_VERTEX_SHADER_BIT,vertexProgram);glUseProgramStages(pipeline,GL_GEOMETRY_SHADER_BIT|GL_FRAGMENT_SHADER_BIT,geomFragProgram);

Binary upload

Compiling and linking shaders, regardless of which method you use, can take a long time. The more shaders you have, the longer this process takes. It is often useful to be able to cache the result of program linking, so that this cached program can be reloaded much faster.

This is done via a set of calls. Given a successfully linked program, the user can fetch a block of binary data, in a certain format, that represents this program. The first step of this process is to get the length of this data by calling glGetProgram​ with GL_PROGRAM_BINARY_LENGTH on the program. Armed with this length, the actual binary can be obtained with this function:

bufsize​ is the maximum number of bytes available in binary​. length​ is an output value that states how many bytes were copied into binary​; it is optional and may be NULL. binaryFormat​ is an output value that specifies the format of the binary data. It is not optional, and it must be stored alongside the actual binary data.

Given the format and the binary data, a new program object can be created with this binary data. This is done via this function:

This function will upload the binary​ data (who's length​ is as given), which is in the given binaryFormat​, into the program​. If the upload is successful, then program​ has effectively had a successful link call performed on it.

This function can fail if binaryFormat​ is not a supported format. You can query the allowed formats with glGetIntegerv​, using GL_NUM_PROGRAM_BINARY_FORMATS to get the count, and GL_PROGRAM_BINARY_FORMATS to get the formats. It can also fail for other reasons; you cannot guarantee that a binary can be loaded.

State of the program

Program objects contain certain state. The program binary only encapsulates the state of the program at the moment linking was successful. This means that all uniforms are reset to their default values (either specified in-shader or 0). Vertex attributes and fragment shader outputs will have the values assigned, as well as transform feedback data, interface block bindings, and so forth.

If glProgramBinary​ is successful, it should result in a program object that is identical to the original program object as it was immediately after linking.

If the original program was separable, then the program built from the binary will also be separable. And vice-versa.

Binary limitations

Program binary formats are not intended to be transmitted. It is not reasonable to expect different hardware vendors to accept the same binary formats. It is not reasonable to expect different hardware from the same vendor to accept the same binary formats.

Indeed, you cannot expect the cached version to work even on the same machine. Driver updates between when the data was cached and when it is reloaded can change the acceptable binary formats. Therefore, glProgramBinary​ can fail frequently. If you use this functionality, you must have a fallback for creating your shaders if the binary is rejected.

Error handling

Interface matching

Shaders in stages have inputs and outputs. Most values output by shaders directly feed subsequent shader stage input variables. There are rules for how these must match.

When linking multiple shader stages together, these rules are checked at program linking time. Therefore, mismatching interfaces between stages are linker errors. However, the interface between separable programs in a pipeline can only be checked at runtime, when the pipeline is used.

Directly linking multiple shader stages together requires that all outputs from one stage are consumed by inputs from the next active stage and vice versa. Failure to do this results in a linker error. However, separable programs do not have exactly match between the separate programs. The results of an inexact match are described below.

Outputs and inputs can either be in an Interface Block or as loose output/input variables. The rules for matching differ between them.

An output interface block matches an input interface block if:

The interface block names match. Remember: the block name is different from the instance name. The block name is the one at the top; the instance name is the one at the bottom. Instance names can be different between stages.

The output block has the same members as the input block. This means that each block contains members that:

Are declared in the same order.

Have the same names.

Have types which match exactly. If they are arrays, the element counts must match.

Qualifier matching

Type qualifiers do not match exactly. Obviously, the storage qualifiers (in​ and out​) need not match. Most other qualifiers must match, but the interpolation qualifiers do not. Only the qualifier set in the Fragment Shader matters. Even if no qualifier is set in the fragment shader, the default of smooth​ and the lack of centroid​ will be used. The interpolation qualifiers are simply irrelevant on all other shader stages.

Array interfaces and shader stages

Certain shader stages automatically aggregate their inputs or outputs into arrays. For example, Geometry Shaders take an array of elements, where each array index represents a vertex. Tessellation Control Shaders take an array of inputs and writes to arrays of outputs, with each index representing a vertex.

For the purpose of determining an interface match, such aggregate inputs or outputs are considered not to be arrays. patch​ qualified outputs/inputs for Tessellation Control/Evaluation Shaders are not aggregated in arrays, so they don't count here. If you make a patch​ variable that is an array, the array index will count.

Separate program matching

Separate programs are allowed to have matches that are not exact (ie: where every output is not consumed by every input).

However, a mismatch can only lead to defined behavior if loose variables are used and those variables use layout(location)​ qualifiers. All other input variables will have undefined values on an non-exact match. This includes particular input variables that themselves match; only those matching using layout(location)​ qualifiers will work.

Therefore:

//Output shaderoutvec4one;outvec3two;layout(location=1)outvec3three;layout(location=2)outvec4four;outOutBlock{vec4five;};//Input shader 1layout(location=1)invec3val1;//Matches with `three`.inivec4one;//Mismatch due to type. UNDEFINED VALUE.invec3two;//UNDEFINED VALUE, due to other mismatch.inOutBlock{vec4five;//Also UNDEFINED VALUE, due to other mismatch.};

Also, when using layout(location)​ qualifiers, the types do not have to exactly match. If the output is a vector type that has more components than the corresponding input (again, correspondence by layout(location)​ rather than name), and the two types match in basic component type (ivec3​ has the basic int​ type), and that basic type is notdouble​, then the two are considered to match. The extra components written by the output are ignored. So it is possible for the following to match:

Validation

A program object, or program pipeline object, must be valid to be used in rendering operations. As much of this validity is checked at link-time as possible; however, some of it references the current OpenGL state. Therefore, some of it must be tested at runtime. For program pipelines, some validity that would have been checked at link-time for non-separable programs (such as interface matching) must be checked at runtime.

The validity of a program or pipeline object can be checked at any time using these functions: