Stripping scriptable shader variants

Massively reduce Player build time and data size by allowing developers to control which Shader variants are handled by the Unity Shader compiler and included in the Player data.

Player build time and data size increase along with the complexity of your project because of the rising number of shader variants.

With scriptable shader variants stripping, introduced in 2018.2 beta, you can manage the number of shader variants generated and therefore drastically reduce Player build time and data size.

This feature allows you to strip all the shader variants with invalid code paths, strip shader variants for unused features or create shader build configurations such as “debug” and “release” without affecting iteration time or maintenance complexity.

In this blog post, we first define some of the terms we use. Then we focus on the definition of shader variants to explain why we can generate so many. This is followed by a description of automatic shader variants stripping and how scriptable shader variants stripping is implemented in the Unity shader pipeline architecture. Then, we look at the scriptable shader variants stripping API before discussing results on the Fountainbleau demo and concluding with some tips on writing stripping scripts.

Learning scriptable shader variants stripping is not a trivial undertaking, but it can lead to a massive increase in team efficiency!

Concepts

To understand the scriptable shader variants stripping feature it is important to have a precise understanding of the different concepts involved.

Shader asset: The full file source code with properties, sub-shader, passes, and HLSL.

Shader snippet: The HLSL input code with dependencies for a single shader stage.

Shader stage: A specific stage in the GPU rendering pipeline, typically a vertex shader stage and a fragment shader stage.

Shader keyword: A preprocessor identifier for compile-time branches across shaders.

In Unity, uber shaders are managed by ShaderLab sub shaders, passes, and shader types as well as the #pragma multi_compile and #pragma shader_feature preprocessor directives.

Counting the number of shader variants generated

To use scriptable shader variant stripping, you need a clear understanding of what a shader variant is, and how shader variants are generated by the shader build pipeline. The number of shader variants generated is directly proportional to the build time and the Player shader variant data size. A shader variant is one output of the shader build pipeline.

Shader keywords are one of the elements that cause the generation of shader variants. An unconsidered use of shader keywords can quickly lead to a shader variants count explosion and therefore extremely long build time.

To see how shader variants are generated, the following simple shader couns how many shader variants it produces:

The total number of shader variants in a project is deterministic and given by the following equation:

The following trivial ShaderVariantStripping example brings clarity to this equation. It’s a single shader which simplifies the equation as following:Similarly, this shader has a single sub shader and a single pass which further simplifies the equation into:

Keywords in the equation refers to both platform and shader keywords. A graphics tier is a specific platform keyword set combination.

This shader variant total is given for a single supported graphics API. However, for each supported graphics API in the project, we need a dedicated set of shader variants. For example, if we build an Android Player that supports both OpenGL ES 3 and Vulkan, we need two sets of shader variants. As a result, the Player build time and shader data size are directly proportional to the number of supported graphics APIs.

Shader build pipeline

The shader compilation pipeline in Unity is a black box where each shader in the project is parsed to extract shader snippets before collecting the variant preprocessing instructions, such as multi_compile and shader_feature. This produces a list of compilation parameters, one per shader variant.

These compilation parameters include the shader snippet, the graphics tier, the shader type, the shader keyword set, the pass type and name. Each of the set compilation parameters are used to produce a single shader variant.

Consequently, Unity executes an automatic shader variant stripping pass based on two heuristics. Firstly, stripping is based on the Project Settings, for example, if Virtual Reality Supported is disabled then VR shader variants are systematically stripped. Second, the automatic stripping is based on the configuration of Shader Stripping section of the Graphics Settings.

Automatic shader variants stripping options in the GraphicsSettings.

Automatic shader variants stripping is based on build time restrictions. Unity can’t automatically select only the necessary shader variants at build time because those shader variants depend on runtime C# execution. For example, if a C# script adds a point light but there were no point lights at build time, then there is no way for the shader build pipeline to figure out that the Player would need a shader variant that does point light shading.

Here’s a list of shader variants with enabled keywords that get stripped automatically:

When automatic stripping is done, the shader build pipeline uses the remaining compilation parameter sets to schedule shader variant compilation in parallel, launching as many simultaneous compilations as the platform has CPU core threads.

In Unity 2018.2 beta, the shader pipeline architecture introduces a new stage right before the shader variant compilation scheduling, allowing users to control the shader variant compilation. This new stage is exposed via C# callbacks to user code, and each callback is executed per shader snippet.

Scriptable shader variant stripping API

As an example, the following script enables stripping of all the shader variants that would be associated with a “DEBUG” configuration, identified by a “DEBUG” keyword used in development Player build.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

using System.Collections.Generic;

using UnityEditor;

using UnityEditor.Build;

using UnityEditor.Rendering;

using UnityEngine;

using UnityEngine.Rendering;

// Simple example of stripping of a debug build configuration

classShaderDebugBuildProcessor:IPreprocessShaders

{

ShaderKeyword m_KeywordDebug;

publicShaderDebugBuildProcessor()

{

m_KeywordDebug=newShaderKeyword("DEBUG");

}

// Multiple callback may be implemented.

// The first one executed is the one where callbackOrder is returning the smallest number.

OnProcessShader is called right before the scheduling of the shader variant compilation.

Each combination of a Shader, a ShaderSnippetData and ShaderCompilerData instances is an identifier for a single shader variant that the shader compiler will produce. To strip that shader variant, we only need to remove it from the ShaderCompilerData list.

Every single shader variant that the shader compiler should generate will appear in this callback. When working on scripting the shader variants stripping, you need to first figure out which variants need removing, because they’re not useful for the project.

Results

Shader variants stripping for a render pipeline

One use case for the scriptable shader variants stripping is to systematically strip invalid shader variants of a render pipeline due to the various combinations of shader keywords.

A shader variants stripping script included in the HD render pipeline allows you to systematically reduce the build time and size of a project using the HD render pipeline. This script applies to the following shaders:

Screenshot of the Fontainebleau Photogrammetry demo using the HD Render Pipeline from the standard PlayStation 4 1920×1080 resolution.

Furthermore, the Lightweight render pipeline for Unity 2018.2 has a UI to automatically feed a stripping script that can automatically strip up to 98% of the shader variants which we expect to be particularly valuable for mobile projects.

Shader variants stripping for a project

Another use case is a script to strip all the rendering features of a render pipeline that are not used for a specific project. Using an internal test demo for the Lightweight rendering pipeline, we got the following results for the entire project:

Unstripped

Stripped

Player Data Shader Variant Count

31080

7056

Player Data Size on disk

121

116

Player Build Time

839 seconds

286 seconds

As we can see, using scriptable shader variant stripping can lead to significant results and with more work on the stripping script we could go even further.

Screenshot of a Lightweight pipeline demo.

Tips on writing shader variants stripping code

Improving shader code design

A project may quickly run into a shader variants count explosion, leading to unsustainable compilation time and Player data size. Scriptable shader stripping helps deal with this issue, but you should reevaluate how you are using shader keywords to generate more relevant shader variants. We can rely on the #pragma skip_variants to test unused keywords in the editor.

For example, in ShaderStripping/Color Shader the preprocessing directives are declared with the following code:

This approach implies that all the combinations of color keywords and operator keywords will be generated.

Let’s say we want to render the following scene:

COLOR_ORANGE + OP_ADD, COLOR_VIOLET + OP_MUL, COLOR_GREEN + OP_MUL.

First, we should make sure that every keyword is actually useful. In this scene COLOR_GRAY and OP_SUB are never used. If we can guarantee these keywords are never used, then we should remove them.

Second, we should combine keywords that effectively produce a single code path. In this example, the ‘add’ operation is always used with the ‘orange’ color exclusively. So we can combine them in a single keyword and refactor the code as shown below.

Of course, it’s not always possible to refactor keywords. In these cases, scriptable shader variants stripping is a valuable tool!

Using callbackOrder to strip shader variants in multiple steps

For each snippet, all the shader variant stripping scripts are executed. We can order the scripts’ execution by ordering the value returned by the callbackOrder member function. The shader build pipeline will execute the callbacks in order of increasing callbackOrder, so lowest first and highest last.

A use case for using multiple shader stripping scripts is to separate the scripting per purpose. For example:

How about also having some more UI-enabled options too (like saying I am not going to use lights of type x/y/z if you can not know the answer) and display the shader counts before/after, so not everything requires editor scripts or at least you can do a little more [quickly] without scripting?

The gains using scriptable shader variants stripping is proportional to the number of shader variants compiled. If compiling a single shader variants is really slow, like on Metal, the gains will be particularly significant.