Baking and Texturing Steps

In the baking and texturing phase:

The info of the high poly mesh is baked into 2D texture maps that can be used on the low poly mesh. This way you get the details of the dense high poly mesh in a much more optimized form that can be used in real time in the game.

Using the baked maps like the normal map and the ambient occlusion maps for example the texturing can be started

Texturing is done by using many materials, masks and generators etc. to achieve the final look that is wanted

Once the textures are ready they are exported from Substance Painter into 2D image files

A PSD file with all the maps combined as layers is created and saved as the work file

Each separate map in Targa (TGA) format is saved to the base SVN and used in the game

Exporting from Modo to Substance Painter

At this point you should have ready a high poly mesh, and a low poly mesh with UVs

In Modo, before exporting the mesh into Substance Painter, apply a named material to it

with the mesh selected, press 'M' and type in a name in the drop down box

Delete all the other materials from the shaders tab, if there are any

The material name should be the same as what your texture filename will be (e.g. sample_asset_mossy). This isn't absolutely necessary, but it will speed things up a bit later on.

Substance Painter's export config will name the files based on the texture set name

In case you have multiple UV-sets, you need to add a separate material for each of them, so you'll get a separate texture set for each. The naming should be the same as above.

Remember to tick Save Materials when exporting

If you use an OBJ file

You can use OBJ in case you don't need any of the parts baked separately

For example if you have just one model or several models that are separate and not touching each other

If you have multiple meshes with different UVs, add each of them their own material in Modo

Merge all the meshes together and export as one obj

If you use an FBX file

Make sure your naming matches with the low poly and high poly

Notice that the ZBrush always has the first letter in caps, so in modo this should also be the case.

Select everything

Export the layers using the same settings when exporting to Zbrush, except remember to additionally select

Save smoothing groups

Save tangent basis

Also recommended Save materials (To get texture sets to work and/or to have properly named layers when exporting the textures.)

Modo Export FBX:

Go to the Modo game tools tab and select the following settings & export the FBX:

Starting a New Project on Substance Painter

The resolution can be changed later without quality loss. Bigger looks better, but can get slow in complex scenes.

Select and add all the texture maps if you baked them externally, if not you'll bake them in Substance Painter

Normal, AO and color ID maps

Adjust the document resolution to the right size

Select DirectX as the normal map format (see image)

For existing projects, or in case you forgot, it can be changed later by going to Edit -> project settings

This means you don't need to flip channels in Photoshop anymore

Baking

You can use xNormal, Substance Painter or Substance Designer for baking the texture maps for the 3D asset (both of these use the same tangent space algorithm for normal maps (mikktspace))

As a rule of thumb use Substance Painter for baking most of the texture maps, xNormal for baking the alpha/opacity map

Baking in the Substance Painter

Baking settings

Even if you have the basic maps baked outside the Substance Painter, you'll need to bake some extra maps that the Substance Painter uses for the mask generators etc

Most of the settings with explanations can be found at the Baking Substance Painter Documentation, but here are some recommendations. See Allegorithmic.com : Baking.

In Texture Set Settings, select 'Bake textures'. Click on the document icon next to 'High poly meshes' and select your high poly. Output size should be relative to your final texture size. Bigger is better, but there is no use baking 4K maps for a 128x128 texture.

You can bake additional maps (based on the normal map) in Painter to help with texturing, remember to adjust the resolution

First bake only the Normal map and test that everything works, if it doesn't work check the Troubleshooting

ID

Color Source: Vertex Color (If you used polypaint to mask mesh parts)

Ambient Occlusion

Self Occlusion: By Mesh Name

Ignore Backface: On

Thickness

Self Occlusion: By Mesh Name

It's a good idea to bake AO and Thickness separate from the other maps and with a lower AA setting, since they take a lot of time to bake

Position

If you want to add a gradient on Y-axis (from top to bottom) to the whole mesh:

Mode: One axis

Axis: Y

Normalization: BBox

Troubleshooting

If nothing can be seen in the maps check that the naming is correct. You could also try to bake without the cage and get better results in some cases.

If your maps show as black or as a weird mess you might have your scale wrong. Import your high and low poly meshes into the same scene and compare them. Marmoset Toolbag is great for this, since it can load the high poly mesh quicker than Modo. See Marmoset.co : Marmoset Toolbag.

If your map has artifacts, holes or other problems, you might need to make a cage model

In case you'll bakes have some weird shading in them, try triangulating those parts or the whole mesh.

Have the low poly mesh triangulated in Modo (shift+T) before baking, but save a non triangulated version too in case the mesh needs to be edited

In case you use a triangulated mesh for baking, the same mesh should be also the one that is used in the game

In any case have a non-triangulated version in the Modo file in case the model will need to be edited in the future

You can keep the triangulated version in the same file on a separate layer etc.

Drag and drop the alpha map from the texture shelf to the Base color button at the Properties - Fill Material panel

Notice that the alpha map will not be working if you check the different channels using "C", it's only visible in the material mode "M" where you can see all the channels combined

Texturing

The Frozenbyte engine now supports PBR (physically based rendering), meaning that what you see in Substance Painter is a good indication of what you will get in game. Not a lot of additional tweaking should be required, apart from some individual special cases. Of course the lighting and post process effects in the game affect the look of the textures.

There are fill layers and layers in Substance Painter

Fill layers are used not only for solid colors but also for adding maps

Layers are used for painting on

In Substance Painter workflow a lot of the things are done by using mask and painting the mask with black or white revealing the color of the fill layer, less of painting actual colors on things

Each layer has color, metal, rough, normal and height channels which can be turned on and off

With "C" you can scroll between different channels in the preview window and with "M" you can see all the channels together

In the top of the Layers panel it's possible to change the mode to Base Color, Metallic, Roughness, Normal, Height

Setting up channels

When creating a project, remember to set 'Normal map format' to DirectX, so you won't have to do any tweaking in Photoshop later on. You can change this later on by going to Edit -> Project configuration.

The Frozenbyte engine supports PBR now, so the default channels should work fine. To add channels, click on the small '+' icon in the TextureSet Settings window, and '-' icon to remove them.

Height, Substance Painter bakes the height map into the normals, so this should be enabled to get most out of the materials even though FB engine doesn't currently support heightmaps/displacement

(Displacement, If you want to preview height/displacement on your mesh, you'll need to add this channel and enable parallax occlusion mapping in Viewer settings. Height doesn't directly convert to displacement, so you'll have to find a workaround.)

Most of the time you will need only 3-4 maps: Base color, Normal, (Metallic), Roughness.

NOTE If you use a larger document size while editing, remember to check the correct size every once in a while. You might lose a lot of detail when you export your final texture size if you only work on the higher resolutions. The switch between resolutions is lossless, so there's no harm in doing it.

Setting the Base Materials

Folders and Layers

The layer structure works similarly to Photoshop, and you can apply a mask to the folder using your baked ID map.

Separate each material area of the character with a fill layer that is put inside a folder & create a mask for that folder that you use to define the area where that material is used

Create this folder with a different color fill layer inside & a mask on the folder for each different material that the model has

You can select the areas from the UVs to mask, but the easiest way is to bake a color ID map for the model based on the high poly model's colors in Zbrush

After baking this color ID map you can use it in painter to mask each area, the colors can be crazy and loud to better use them for masking

How to add the base colors

In the Layers window add a fill layer by clicking the button on top of the Layers window

In the Layers window add a folder by clicking the button on top of the Layers window

Drag and drop the fill layer into the folder

Name the folder and layer, for example folder: skin/hair/metal/etc, layer: base color

At the Fill window adjust the Base color you want to use or use color picker to pick the color from anywhere on the screen

You can also adjust the overall Metallic and Roughness from the Properties - Fill window

Masking

with the fill layer selected, click the Configure mask button on top of the same Layers window

Choose the Add mask with color selection, this let's you use the color ID map you have assigned in the TextureSet Settings Window

At the Properties - Color Selection decide what area of the model you want to pick for this layer by clicking the Pick color and use the eye dropper to pick the color from the model or the UV texture map

Raising the tolerance value gets rid of gaps on the mask edges, but too high values makes the mask bleed to other colors

The end result is a mask based on the color in the color ID map. In this example the rest of the mesh is grey and the area inside the mask is red as the mask prevents the red color filling the whole mesh with red. Now you can change the red into any color you want and add layers and generators and smart materials etc. to get the material done.

Once you have your folders set up and masked, you can add materials and paint as you like.

Smart Materials

You can download user made smart materials, masks, etc. from Substance share website, or make your own.

Smart materials are basically layer stacks contained in folders, so it's easy to make your own.

Once you have created a material by layering base materials, effects etc, group them up in a folder (select all and Ctrl + G).

Name the folder, right click it and select 'create smart material'. You'll have a new smart material with the name of the folder you just made, with all the layers and settings intact and saved, which you can use freely in other projects.

Name a Fill layer "Opacity" etc. and keep it at the top of the layer stack to maintain the opacity

Experiment with adding more filters to the mask stack in addition to the color selector.

You can do a lot of the same stuff as with regular layers (blending modes, texture fills, etc.), and can achieve some complex results.

Normals

After adding all the maps to the project in the TextureSet Settings panel there might be the issue that the normal map is not showing (you can press C until you see the normal map to see if it works)

You'll need to add a fill layer and drag and drop the normal map from the Shelf -> Textures to the Properties - Fill Normal channel. Turn all the other channels off for this layer from the same panel

In the TextureSet Settings use the setting Normal mixing: Replace

If you want to paint on the normal map, create a new layer and go to the Normals mode in the Layers panel, select Normal layer style to paint over. The default "NMdt" layer style means that it mixes the layers' contents but the "Normal" layer style goes on top of everything

Adding a custom shelf in Substance Painter

How to add a custom Shelf

Edit -> Settings -> Shelf

Add Name & Path and click the plus button & OK

Now you should be able to find the materials in that folder every time you use Substance Painter and if new materials are added there, they should appear automatically

Exporting

Once you have finished texturing, go to File > Export textures, or Ctrl+Shift+E, or right click on the scene and choose Export Textures

In the export dialogue, change the texture size to the size you want the final texture maps to be

Point the file path to the right folder location

File format: Up to you which one to use, either one is easy to convert to the other

Psd (Will export all the maps into one psd-file as layers)

Targa (You'll get all the maps as separate textures, ready to use')

Document channels + Normal + AO

Use the base color, metallic, roughness and normal_directX

When the final resolution is small (say, 1024 or under) it is better to have the working resolution larger than the export resolution, instead of down scaling the working resolution and exporting that

This results in smoother textures, since the materials are generated at a high resolution and then down scaled to the export resolution

Doing this at high resolutions is a bad idea since 2k and above working resolutions are a pain in the ass to work with, and the downscale advantage will be close to nonexistent at this level.

The texture files that go to the game's base SVN and into the game need to be in the Targa (TGA) format

Other maps: Now that the engine supports PBR, you should get the same results from Painter and editor, but in some cases some tweaking in Photoshop might be needed and it might be a good idea to export the ID map and use that as an aid. You can export it along with the rest of the bakes by selecting "additional maps" from the painter export dialogue.