Gl_NormalMatrix replacement in shaders

First let me apologize for asking what I am sure is an overly asked question but all my searching and testing has resulted in nothing.

I am attempting to remove all the GLSL built-ins from my shaders and the last one for #version 150 is gl_NormalMatrix. From all my reading it’s the transposed inverse of the upper 3x3 of the product of the Model and View matrices. So I have tried numerous permutations of this line (mat3 glNormalMatrix = transpose(inverse(mat3(mView * mModel)))) with no success replicating gl_NormalMatrix.

Below is a normal mapping vertex shader and it works perfectly with the gl_NormalMatrix built in but this built in is depreciated in version > 120 thus the exercise.

I am making some assumptions here but I am pretty certain may matrices are correct and I pass a projection, view(camera) and model(world) matrix in to transform the object coordinates. Additionally I have annotated what space I believe the vec3 to be in and what direction the mat4 is intended to transform to. I don’t think it’s a problem with my TBN but not ruling that out also. I just assumed I’d be able to calculate a ‘replacement’ for gl_NormalMatrix from the components passed in but that is proving to be harder than I had suspected. The offending line is in the CalculateTBN near the bottom with the working line uncommented and the ‘what I think is correct’ commented.

I don’t rule out the TBN calculations but I believe they are correct as they work with the gl_NormalMatrix.

If seeing the fragment shader or my TBN calculations will help I will post but I suspect those components may be a side matter to this activity. Also this shader is not optimized I know but I’m following the mantra, make it work (built-ins), make it work right (remove built-ins), make it fast (move some or most of the matrix math back to CPU). Maybe that exercise will solve this problem but I suspect I should be able to replicate a replacement for gl_NormalMatrix in the shader also. Additionally any insight into making it better is always appreciated.

Notes on what’s happening when rendering a plant with normal maps. With gl_NormalMatrix the lighting seems correct and stable to light source. Normal map looks correct, specular stay with camera angle. With my attempts to replicate the gl_NormalMatrix the dark side rotates with the globe, specular does odd stuff like going opposite directions, culminating at points on the globe and other odd effect. The normal mapping seems to be correct as it still looks accurate but with the shaded and specular moving it can be hard to tell.

If you are only doing uniform scaling, AFAIK mat3(nView*nModel) will work; but it is more normal to calculate it on the cpu and pass it as another unform. Also I would not be inclined to use variables names of the form "gl_xx" for my own variabes.

I will say I am thrown by the term 'uniform scaling'. What are you referring? The mView is the camera's matrix and the mModel is the model's matrix. Both are, on the CPU side, 4x4 matrices for rotation, scaling(of which i am not scaling) and translations. Camera matrix is all negative values of position/orientation and multiplication is reversed but seem to be compatible with the gl_Position calculations and the inverse(mModel) calls to transform camera an light 'world' positions into object space.

When it comes to uniform scaling, I have also seen reference to orthogonal rotation matrices which I assume is X = Y = Z type of rotations which is not what I have here. I am actually building the required matrix from a Quaternion for orientation and a Vector3 for position.

As far as the gl_XX variables names I agree. I assume your referencing the glNormalMatrix variable in the CalculateTBN call. I used that name as a quick replacement for the gl_NormalMatrix just to be lazy in the code edits. My plan is to optimize this my removing all the unneeded calculations like the inverse(mModel) and (if I can figure it out) the glNormalMatrix to the CPU side. I may even go as far as passing in both a_vTangent and BiNormal(BiTangent). BTW...the a_ is for attributes, and thinking about moving to u_ and v_ for uniform and varying respectively but it appears that those may have also been depreciated to in/out).

So...I tried as you suggested, and as I believe I have, and the results are similar to before. The lighting seems to be bound to the model and camera position seems to have some influence...

Could something be transposed that I am not aware of and it getting washed out in the calculations? I'd suspect no but I am kind of blind when it comes to the inner workings of shaders.

Thanks for the response...this is a tough one for me...any thoughts on how to extract the gl_NormalMatrix values from the shader for examination. If I could see it I should be able to reverse it and then replicate it.

OK...so more research on orthogonality has delivered this. My upper left 3x3 of my ModelView matrix is orthogonal. All rows and columns have a length of one or as close as it will get. The transpose of the ModelView equals the Inverse of the ModelView so I think I have answered the question of uniform scaling if indeed that is what you were getting at.

So...Is it my TBN? I cannot say. If works with the GL supplied normal matrix. All I can assume is there is some piece of magic I am missing. If I could extract the gl_NormalMatrix values from the shader I could attempt to reverse this and get an idea of whats wrong.

I did some testing with vDebugColor = gl_NormalMatrix[x][y] and passed through to fragment shader and multiplied to final color on output. Went through all 9 iterations and 0,0; 1,1; 2,2 = 1 other = 0. Where it was 1 I saw my model rendered in perfect color, where it was 0 the model was black, rendered but black.

So I set my variable glNormalMatrix to mat3(1) and it worked. Took gl_NormalMatrix out all together and it worked.

So now the question. Whats up? Why would gl_NormalMatrix be mat3(1)? Also if that's the case then I believe it's telling me my tangents are in object space not tangent space. I suspect this as my normals are an attribute and in object space. The matrix created uses normal(attribute), tangent(attribute) and a cross(assume in object also). Thus my calculations to convert to tangent space are wrong.

Anyone ever seen gl_NormalMatrix = mat3(1)? Any reason this may be the case?

BTW...I've read the lighthouse tutorial and many others backwards and forwards. I never suspected an identity matrix.

I would love to hear someones explanation of why gl_NormalMatrix could be identity.

gl_NormalMatrix doesn't know what a tangent or bitangent is. That's not what it's for, so it being identity means nothing about your tangents or whatever. It's just the inverse/transpose of the upper 3x3 portion of the modelview matrix. That's all.

If it's identity, then odds are good that gl_ModelViewMatrix is also identity.

Wait a second...since this is all shader based and I am not using glMatrix...() then your correct, everything is identity.

I am attempting to emulate OpenGL ES on the desktop. So I set GL_PROJECTION and GL_MODELVIEW to identity so the fixed pipeline doesn't get in the way. Since I want to eventually move to a mobile platform, I know there will be other shader challenges there, I wanted to attempt to get the fixed pipeline out of the way. Since there appears to be no ES mode for the desktop GL I figured identities would cause nothing to happen. That's probably whats happening.

How would one get the desktop GL to stop using the fixed functionality? This is a quick development setup so I am using C# and OpenTK with TAO compatibility for easy port back to C++ later. The only real OpenTK I am using are the Math libraries and the GL control for windows forms. Everything else I am trying to keep pure C# and only the GL calls required like glActiveTexture, glSetUniform, etc... and the like.

I never thought to consider the identities for projection and modelview I set at the start of the program. It was just a way to get the fixed functionality to be quiet.