So I'm back at improving the terrain texture blending visuals. As some of you may remember, I worked on this technique a long time ago and got stuck because I lost the height information after each terrain layer.

So, first a bit of background information:

Torque's Terrain Texture blending
In general, Torque's terrain is pretty well done for what it seeks to accomplish. It lacks dynamic tesselation and what not, but it gets the job done for most games. T3D has a "terrain" object, which has a huge diffuse map which is used for coloring the terrain. Up-close you can mix it up with detail textures to allow detail into the over-all coloring of the terrain.

The terrain can have 32 different texture layers, which consists of:

A Diffuse Texture - Which contributes to the aforementioned diffuse map

A Detail Texture - Which provides up-close detail

A Macro Texture - Which provides medium-distance detail

A Normal Texture - Which provides normal and height information for the detail texture

Furthermore, the terrain is split into "chunks" of smaller size, that may be occluded in order to allow for better LOD'ing of the terrain and provide performance improvements. Each of these chunks are rendered consecutively.

T3D currently blends these linearly, using a layer-texture which holds the information about what texture is painted to what layer. This is similar to using a lerp function to blend between two variables. The result of blending a red, a green, a yellow and a black texture can be seen in the following image:

Which would typically provide results similar to the following image, when used on actual textures:

And to if we lerp-blend left-to-right between two textures it looks like this:

The sampler limit
When rendering each layer, we need to send 4 textures per layer to the shader. There is an inherent limit in how many registers each shader can receive, which varies from shader model to shader model. I believe it is 32 for D3D11, however I don't feel like finding a source on that again. However, assuming it is 32, we can easily see how we might hit that cap quickly:
$$\frac{32}{4}=8$$
I do believe the cap was even lower for D3D9, so at some point, someone decided that we would send just 3 textures down to each pass over the terrain, meaning a terrain block can be rendered over 11 draw-calls.

My first attempt
As mentioned, I worked on this a long time ago. Back then I encountered an issue with these multiple passes made over the terrain (the 11 draw-calls), namely that I lost the height information between each pass. Now this is bad, since the approach I took was a height-based blending of the textures, so I looked at flip-flopping a height-map render target.

What does flip-flopping mean? The concept is pretty simple actually. Due to how the GPU works, we can't read and write to the same texture each pass. If we need to both read and write the height information to a texture, then we need to create 2 textures, and swap them around for each pass. So in pass 1 we read from texture 1 and write to texture 2, in pass 2 we read from texture 2 and write to texture 1, in pass 3 we read from texture 1 and write to texture 2, etc. This process is called flip-flopping.

However, I never really got flip-flopping to work, for some reason it just wouldn't work.

But what now!?
It works! I dunno how or why, possibly the change to D3D11 changed something, but nevertheless it works!

Using his technique on the left-to-right example showed earlier, we get the following result:

And it can be easily customized to change the results, by changing the "blendDepth" variable:

And the in-game example looks like this:

What's left?
Confirming that it works. There's a bit of spaghetti code, especially in the generated shaders, I will have to clean them up, and try some more use-cases. Furthermore, some performance tests are necessary to see how much the new shader code and flip-flopping of textures impacts the performance of the terrain rendering system.
So I can't promise you that this will see the light of day or be usable, but I think it is pretty cool, and I hope it can help us take some steps in the direction of more nuanced and realistic texture blending in our terrain rendering.

As far as the 11 barrier, that was a dx9 limitation for communicating between vertex and pixel shaders. Why we embedded https://github.com/GarageGames/Torque3D ... al.cpp#L40 (though I could see say, having a cap for the number of passes based on renderer if we end up in a situation where dx11 and opengl end up with wildly different caps.)

Monkeychops: He's simplifying the explanation a bit. If you follow the trail of that sgMaxTerrainMaterialsPerPass you'll note that it blends multiple passes. Terrain Layers themselves look to a rough equivalent of a splatmap, cluster up to (currently hardcoded 3) materials in one render, then layers over the result itteratively untill it runs out of new layers.

Seems to work nicely here.
For some reason the T3D terrain textures always look really over-saturated and blurry to me though compared to the original. That's always been something that has put me off the engine.

Monkeychops
Then lessen the saturation. The problem with terrain textures is, that you have 2-3 layers that get blended into each other, which can produce high saturations on some places, so you have to work with very low saturations/brightness variations.
Terrain detail textures should not have any color saturation at all, or very little, since it will get added to the base texture saturation, which obviously will produce an oversaturated result, if not taken care of before.

However, the version I put up here, still has a few issues blending-wise that might affect the final result. I'm looking to upload a version with a more correct implementation of Andrey Mishkins' technique. We will see how that affects the colour of the terrain textures.