I'm using a technique I discovered in a paper to determine the normal to the terrain using the heightmap data that sets the height of each vertex in a 2D plane. The problem is that the shading is still totally flat and the normals still point directly up, as they do in the original plane. Can anyone see why? Should I be doing the normal calculation in the pixel shader (which I also tried, doesn't work either).

Yeah, I tried outputting the normal and it gives me a totally green colour across the terrain, i.e. (0, 1, 0), just like on the original plane. Yes, I checked the ambients and light power too. I'll have another mess around with it!

Have you tried to use the normal without multiplying it by the world matrix? Perhaps your world matrix isn't properly being set, which could lead to issues. Also, why have you chosen to use tex2Dlod for sampling the texture? It should still work fine, but I was just wondering why you would go that way...

I'm by no means an expert. I don't think you need to calculate normals in the shader unless you're terrain is moving every frame, as opposed to the camera moving every frame. Below is the code I use to get the terrain normals when I initialize new terrain, in C#, which seems to work. It's not my code, You could probably find a better algorithm for generating the normals, since it doesn't run every frame speed is not a huge factor.

As far as I know, tex2Dlod is the right function for texture sampling in a vertex shader, there are restrictions in which sampling functions are supported.

I don't think you need to calculate normals in the shader unless you're terrain is moving every frame,

You are right that normal recalculation isn't necessary in the shader, but on the other hand, it allows you to use GPU to calculate something that typically you'd need to do on CPU so, this way you'll reduce CPU load / storage load and also you'll need to send less data to the GPU. The normal calculating function presented here was used in certain Battlefields games, so I guess it has some advantage too. Of course, you'll need to have the height map available for the normal calculations.

You are right that normal recalculation isn't necessary in the shader, but on the other hand, it allows you to use GPU to calculate something that typically you'd need to do on CPU so, this way you'll reduce CPU load / storage load and also you'll need to send less data to the GPU. The normal calculating function presented here was used in certain Battlefields games, so I guess it has some advantage too. Of course, you'll need to have the height map available for the normal calculations.

But you would make it only once on the CPU, wouldn't you? As opposed to every frame on GPU. (If the terrain is not changing somehow during time.)

You need to send less data to the GPU, that's right. But that would probably require some profiling, whether it's better to be sending less data and calculating some on the GPU or to be sending more data and spare some GPU time.

Well, as I told, the method has been used in a AAA level game, so I guess that they did the profiling. Of course hardware changes and everything changes, so you can always redo the profiling.

With CPU calculated normals, you'll still do texture read in the shader and probably some unpacking depending on the texture format. With the GPU version you'll do 4 texture reads, but probably they won't be much slower. The amount of ALUs in the GPU version is so small that it won't change much.

So, probably calculating the normals on the GPU won't be any slower than the CPU version.

Well, as I told, the method has been used in a AAA level game, so I guess that they did the profiling. Of course hardware changes and everything changes, so you can always redo the profiling.

With CPU calculated normals, you'll still do texture read in the shader and probably some unpacking depending on the texture format. With the GPU version you'll do 4 texture reads, but probably they won't be much slower. The amount of ALUs in the GPU version is so small that it won't change much.

So, probably calculating the normals on the GPU won't be any slower than the CPU version.

Cheers!

Tom KQT has it right here - if you terrain isn't changing over time, then you should just calculate the normal on the CPU and store it with the terrain vertices. Otherwise you are recalculating the same thing every frame on the GPU, which is a huge waste of computation and bandwidth. If there is a AAA game that does this, I would be interested to know which game you are referring to - maybe their terrain was destructible or something like that...

Did you see my question from above - have you tried to remove the world matrix from the calculation to see if it helps?

Sorry for not replying and thanks for the ideas everyone, since my terrain is static and isn't changing per-frame, I think the best idea may be to create another texture-based container to store the normals which can then be looked up in the shader. I'll see how it goes

Well, as I told, the method has been used in a AAA level game, so I guess that they did the profiling.

Just a small note on this. Don't take AAA games in general as your holy grail as far as technology is concerned. Sure, there are some developers that push the boundaries further and produce extremely optimized and technicaly advanced games. But there are plenty of AAA games that are made in very short time and with focus on other elements, not optimizations.

For example, I think it was on this forum - somebody showed how incredible hight was the amount of draw calls in the early version of Civilisation V. They had a separate draw call for every single icon on the screen, including all those food/wealth/production icons on each tile (if you know the game). That probably won't be the best way, will it? And btw I think this changed in the later versions of the game.

The draw call problem with CIV 5 and terrain normal calculating are two quite different thing, the first is caused by artists not having optimized engine for the purpose and second by programmers optimizing certain aspects of the game engine.

I agree that a certain amount of criticism is always good when learning new things. The normal calculating method was described in the engine documentations which was about_new tech and new ways of doing things, not in the hobby sections of some discussion area. My point here, those people publishing things on the net have quite solid understanding of what they are writing.

Indeed, in the later versions of the game, they changed the terrain rendering code and what is stored in textures etc. But once again, it was a question needs. The BFBC2 terrain rendering could go only so far. They improved / changed the code because mixing so many terrain layers became prohibitive in the sense of performance. I used exactly the same code earlier in my hobby projects and I was pleased in it's performance. However, later my needs became different and I adopter another method where I couldn't use the code anymore. Anyhow, a GPU, when correctly used has pretty much computing power and few used ALUs won't affect the performance pretty much at all.