Tuesday, 22 April 2014

Overview

HPL3 is our
first engine to support both PC and consoles. To make it easy to support multiple platforms and
multiple shading languages we have decided to use our own shading language called HPSL. Shader
code written in HPSL goes through a shader parser to translate it to the
language used by the hardware.

The shader
written in HPSL is loaded into the engine at runtime, the code is then run
through a preprocess parser that strips away any code that is not needed by
the effect or material. After that the stripped code is translated to the
language used by the hardware (GLSL #330 on PC and PSSL on the PS4) and then
compiled.

HPSL uses
the same syntax as the scripting or engine code. HPSL is based on GLSL #330 but some of the declarations are closer to HLSL.

// Example
code

@ifdef UseTexture

uniform cTexture2D aColorMap : 0;

@endif

void main(in cVector4f
px_vPosition,

in
cVector4f px_vColor,

in
cVector4f px_vTexCoord0,

out
cVector4f out_vColor : 0)

{

cVector4f vColor = px_vColor;

@ifdef UseTexture

vColor *=
sample(aColorMap, px_vTexCoord0.xy);

@endif

out_vColor = vColor;

}

//Preproccess step

void main(in cVector4f
px_vPosition,

in
cVector4f px_vColor,

in
cVector4f px_vTexCoord0,

out
cVector4f out_vColor : 0)

{

cVector4f vColor = px_vColor;

out_vColor = vColor;

}

// Translation step

#version 330

#extension GL_ARB_explicit_attrib_location
: enable

in vec4 px_vColor;

in vec4 px_vTexCoord0;

layout(location = 0) out
vec4 out_vColor;

void main()

{

vec4 px_vPosition = gl_FragCoord;

bool px_bFrontFacing = gl_FrontFacing;

int px_lPrimitiveID = gl_PrimitiveID;

vec4 vColor = px_vColor;

out_vColor = vColor;

}

Preprocessing

All the
shader code used in SOMA is handwritten. In order to keep all the relevant code
at the same place and to be able to quickly optimize shaders HPL3 uses a
preprocessing step. This has been used for our
previous games as well. A preprocessor goes thorugh the code and removes large
chunks that are not needed or used by the effect or material. The lighting shader
used in SOMA contains code used by all the different light types. Changing a
preprocess variable can change a light from a point light to a spotlight or can be
used to enable shadow mapping. The preprocessor strips blocks of code that are not used, this increases performance since code that has no visual effects is removed completely. Another feature of the preprocess parser is the ability to change the value of
a constant variable, this can be used to change the quality of an effect.

// SSAO
code

for(float d = 0.0; d <
$kNumSamples; d+=4.0)

{

// perform SSAO…

}

The preprocessor makes it easy to do complex materials with multiple textures and shading properties while only performing the heavy computations for the materials that need it.

Translation

After the
preprocess strips the code it is ready to get translated. In the first step all the
variable types and special functions are converted to the new language. Then the main entry function is
created and all the input and output is bound to the correct semantics. In the
last step the translated code is scanned for texture and buffers that get bound
to the correct slot.

Compilation

The
translated code is then compiled. If a compilation error occurred the
translated code is printed to the log file along with the error message and
corresponding row for easy debugging.

Summary

In order to
deliver the same visual experience to all platforms and to make development
faster we decided on using our own shading language. The code is translated to
the language used by the hardware and compiled at runtime. Supporting other shading languages in the future will be very easy since we only need to add another converter.

Modders will still
be able to write shader code directly in GLSL if they chose to.

HPSL Reference

Syntax

HPSL uses
the same syntax used by the scripting language.

Variable Type

Description

int

32 bit signed integer

uint

32 bit unsigned integer

bool

Stores true or false

float

32 bit float

double

64 bit float

cVectorXf

Vector of floats

cVectorXl

Vector of signed integers

cVectorXu

Vector of unsigned intergers

cMatrixXf

Square float matrix

cMatrixXxXf

Non-square matrix (Ex cMatrix2x4f)

cBuffer

Container of multiple variables
that get set by the CPU

Texture Type

Description

cTexture1D

Single dimension texture

cTexture2D

Standard 2D texture

cTexture3D

Volume texture

cTextureCube

Cubemap texture

cTextureBuffer

A large single dimension texture
used to store variables

cTexture2DMS

A 2D render target with MSAA
support

cTextureXCmp

A shadow map texture used for
comparison operations

cTextureXArray

Array of cTextureX textures

A texture
contains both the image and information about what happens when it is sampled.
If you are used to OpenGL/GLSL then this is nothing new. DirectX uses a
different system for storing this information. It uses a texture for storing
the data and a separate sampler_state that controls filtering and clamping.
Using the combined format makes it easy to convert to either GLSL or HLSL.

Textures
need to be bound to a slot at compilation time. Binding is done by using the
“:” semantic after the texture name.

//bind
diffuse map to slot 0

uniform
cTexture2D aDiffuseMap : 0;

Variable Type Modifier

Description

uniform

A variable or texture that is set
by the CPU

in

Read only input to a function

out

Output of a function

inout

Read and write input and output to
a function

const

A constant value that must be
initialized in the declaration and can’t be changed

Entry Point and Semantics

The entry
point of a shader program is the “void main” function. Input and output of the
shader is defined as arguments to this function. The input to the vertex shader
comes from the mesh that is rendered. This might be information like the
position, color and uv mapping of a vertex. What the vertex shader outputs is
user defined, it can be any kind of information that the pixel shader needs.
The output of the vertex shader is what gets sent to the pixel shader as input.
The variables are interpolated between the vertices of the triangle. The input
of the pixel shader and the output of the vertex shader must be the same or
else the shaders won’t work together. Finally the output of the pixel shader is
what is shown on the screen. The pixel shader can output to a of maximum 4
different render targets at the same time.

Some of the
input and output are System defined semantics. System Semantics are set or used
by the hardware.

System Semantic

Description

Type

Shader Type

px_vPosition

Vertex position output. Pixel
shader input as screen position. This is required by all shaders

cVector4f

Vertex (out), Pixel (in)

: X

Output color slot, where X must be
in the range 0-3

cVector4

Pixel (out)

vtx_lVertexID

Index of the current vertex

int

Vertex (in)

vtx_lInstanceID

Index of the current instance

int

Vertex (in)

px_lPrimitiveID

Index of the triangle this pixel
belongs to

int

Pixel (in)

px_bFrontFacing

Indicates if the pixel belongs to
the front or back of the primitive

bool

Pixel (in)

Input to
the vertex shader is user defined. HPL3 has a few user defined semantics that work with our mesh format.

Functions

There are
some functions that are different from GLSL. This is to make it easier to
support HLSL and PSSL.

Arithmetic Function

Description

mul(x, y)

Multiplies two matrices together
(multiplying by using * not supported for matrices)

lerp(x, y, t)

Interpolates between two values

Texture
sampling use functions specific to the HPSL language.

Texture Function

Description

sample(texture, uv)

sample(texture, uv, offset)

Samples a texture at the specified
uv coordinate. Can be used with an integet offset

sampleGather(texture, uv)

sampleGather(texture, uv, offset)

Samples a texture but returns only
the red component of each texel corner

sampleGrad(texture, uv, dx, dy)

sampleGrad(texture, uv, dx, dy,
offset)

Performs texture lookup with
explicit gradients

sampleLod(texture,
uv, lod)

sampleLod(texture,
uv, lod, offset)

Samples the texture at a specific
mipmap level

sampleCmp(texture, uv, comp_value)

sampleCmp(texture, uv, comp_value,
offset)

Performs texture lookup and
compares it with the comparison value and returns result

load(texture, position)

Gets the value of a texel at
the integer position

getTextureSize(texture, lod)

Returns the width and height of
the texture lod

getTextureLod(texture, uv)

Gets the lod that would get
sampled if that uv coord is used

getTextureLevelCount

Gets the number of MipMap levels

It is also
possible to use language specific code directly. Some languages and graphic
cards might have functions that are more optimized for those systems and then
it might be a good idea to write code specific for that language.

5 comments:

Nice idea to write a higher level shader language which translates to different graphics API shader languages for portability. Did you write the parser from scratch or is it built on top of another parser engine (and only define grammar)?

I like your tech articles, very informative - and your games of course (played all three Penumbra games the last easter days, next to be played is AAMFP). Keep up the good work! :D I slowly become a Frictional Games fan. ^^

Btw could you also write a tech article about Occlusion Culling / Visibility Determination (and maybe Content Streaming) in HPL3? As I've read, you now support indoor and large outdoor scenes and I am interested in which techniques you use for both scene types and how you manage seamless transition between indoor and outdoor.

HPL1 uses portal culling and HPL2 CHC, right (Wiki says that ;D)? Do you use any preprocessing in your engines like generating a PVS or anything like that?

Thanks.The parser is built from scratch. The preprocessor part has been in the engine since HPL2 but the translation part is new for HPL3.

HPL3 uses CHC culling just as HPL2. There have been some improvements to it and Im going to add some more improvements later, so i might make a blog post about that. The only thing we bake is static geometry and physics, but that is baked when you start a map the first time (it has been prebaked when the game is released)

I read on Thomas' twitter that you're thinking about supporting DirectX or running on Xbox One. Is that the reason you're mentioning HLSL in the article? Would DirectX be solely for Xbox or would that be an option on PC, too?