Why you should NOT trust 3ds Max's viewport normal-map display!

OK, sensationalist headline. But it got your attention! This was a spinoff from another thread, I thought it deserved its own.

Here are my findings based on Max vs. Maya baking using the native tools and displaying the results in the viewport using the native tools.

Summary:Unless your game engine is set up to use the same tangent basis as 3ds Max's renderer (not it's viewport), your normal-mapped models will never look 100% correct!

I have just done some simple tests tonight, and I can confirm that Max's "baking" tangent space is the same as it's "render" tangent space (no surprise, since the scanline renderer is used to bake normals in Max), however the "render" tangent space does not match the tangent space that it uses in the viewport. This strikes me as just plain insane. It means you'll never get the same result between rendering and viewport display.

If you use a "correct" normal-map for viewport display in Max, then it will look messed up in render, and vice versa... horrible.

I baked the same objects in Maya, and they are displayed perfectly in the Maya viewport. Interestingly enough, this map also looks perfect in Max's viewport! That means that the viewports in both apps derive tangent basis in the same way. However it also means that Maya's baking renderer correctly uses this tangent space! Which makes sense, of course.

It basically seems like since the viewports (and Maya's baker and Mental Ray, by extension) seem to all use the same tangent basis derivation, this is the most "correct" one, and therefore 3ds Max's renderer and baker are "wrong". This should be a standard that all applications adhere to, I see no reason why it shouldn't be.

If you're feeling intrigued you could download those normal maps and flick back and forth between them in an image viewer to see the main differences. You should be able to see them clearly enough here, though.

So, out of interest, does anyone here work in a pipeline where they bake their normal-maps using 3ds Max, and if so, do you know what your game does with them? Have you ever had any problems?

EDIT: ADDENDUM!

Here are some direct links to some of the more useful posts from later in the thread:

Did you flip the green channel? I take my bakes into Photoshop and invert the green...

You shouldn't have to invert the green channel in Photoshop, there is a setting for that in the Render To Texture "Options" dialog. Just set it to the correct Y orientation before you bake.
Every application has a control for this (in Maya it's based on whether the mesh's "tangent space" attribute is set to Left-Handed or Right-Handed), if you're inverting the green channel in Photoshop then you're not baking it right

Second question: Does the same thing occur when using Mental Ray in Max?

I have only tried the Scanline renderer so far. I will try Mental Ray on the same objects, hopefully it will give better results (although that also means more inconsistent results within Max...).

Edit: After trying Mental Ray inside Max (added image to original post), while it is slightly different to the scanline bake, it is still far from being correct, if we're treating the Maya bake as the "most correct" bake.

You shouldn't have to invert the green channel in Photoshop, there is a setting for that in the Render To Texture "Options" dialog. Just set it to the correct Y orientation before you bake.
Every application has a control for this (in Maya it's based on whether the mesh's "tangent space" attribute is set to Left-Handed or Right-Handed), if you're inverting the green channel in Photoshop then you're not baking it right

I pretty much bake all my normal maps with Max. I've thrown the straight Max normals into Source and Cryengine 2 and haven't once had any problems like this. Then again, I'm also mainly doing lots of hard surface and mechanical stuff.

It really does look like the green channel needs flipping, and you should try using Xoliul's max shader, since the max default doesn't display normal maps correctly.

1. Which application is using the "best" tangent basis derivation? As far as I can tell, Max's renderer tangent basis is the "odd one out" - more apps like Maya, XNormal, maybe others are consistent between each other, yet Max isn't even internally consistent!
2. Why is Max's renderer tangent basis different to it's viewport tangent basis? This essentially means that if your normals look correct in the viewport, then they will look wrong in the render. Conversely, if they look correct in a render, they will be wrong in the viewport. This is ridiculous!

3. If you can write an FX shader that uses Max's "renderer tangents" in the viewport, why doesn't Max's default internal shader and supplied FX files use this method? Both Max's "Show Hardware Map in Viewport" option, and it's supplied external shader files like "StandardFX.fx" use the default viewport method, which means any bakes you do in Max will look crappy using these shaders! Insane!

Mop I find it odd that you are posting this now, I thought you used to be a Max user then had to go to Maya. I'm wondering if this happened in older versions of Max or is it an upgrade that came with 2010. I mean they did change how you could shift drag things to extrude edges, so maybe they thought it was a good idea to update the viewport display and not do so with the scanline renderer so now they don't match.

I'm also wondering if this is the case then how is it that you can make normal maps in Max and use them in unreal ed so they look correct or does unreal for example have code that can render this out properly because a coder took the time to do so.

I'm also wondering if this is the case then how is it that you can make normal maps in Max and use them in unreal ed so they look correct or does unreal for example have code that can render this out properly because a coder took the time to do so.

(and eld's point, added the quote in an edit)

Well yeah, I was wondering that too, since I thought they baked stuff in Max?
That either means they've set up the Unreal Engine to use Max's renderer tangent derivation (meaning maps baked in any other app will look wrong!), or they're getting dodgy looking stuff in-game and dealing with it by adding geometry till the problem goes away... or... they just don't care that much

Mop I find it odd that you are posting this now, I thought you used to be a Max user then had to go to Maya. I'm wondering if this happened in older versions of Max or is it an upgrade that came with 2010. I mean they did change how you could shift drag things to extrude edges, so maybe they thought it was a good idea to update the viewport display and not do so with the scanline renderer so now they don't match.

Nope, it's been like this as long as I've used Max. Definitely since max 8, and it was in max 2008 and still a problem in max 2010.

uhm I think even using custon FX in max doesn't save you from artifacts. Afaik the scanline baker bakes for the scanline renderer, which uses different kind of interpolation than what realtime rendering does (not just per-vertex tangent basis). At least that's what I read. One would have to modify the baker in max to get perfect matches. I also remember people here saying that some engines managed to load stuff artifact free in games, but had to do something "special" what that is, they couldn't tell due to NDA stuff.
how that is nowadays I don't know, but last time I checked it was still easy to create tangent maps that only look perfect in scanline.
If I recall correctly then ASE used in unreal import also doesn't encode tangent stuff, which means they indeed must do things the same way max does when importing.

I pretty much bake all my normal maps with Max. I've thrown the straight Max normals into Source and Cryengine 2 and haven't once had any problems like this. Then again, I'm also mainly doing lots of hard surface and mechanical stuff.

It really does look like the green channel needs flipping, and you should try using Xoliul's max shader, since the max default doesn't display normal maps correctly.

You're also adding 5000 hard edges to your models to get a correct result in max, you could take those same models with totally smooth normals in maya and bake them with results that are *near flawless*.

Mop how did you notice this? Did you start writing hlsl shaders or something and noticed the difference? I imagine since unreal is using hlsl for their shaders it's the same, so maybe they have a different baker. Maybe some of epic guys can say something about this.

Sage: I noticed it first, a long time ago, when I was working purely in Max. I thought for a while that it was my lowpoly geometry causing the issues. Then after working with XNormal then Maya for a bit, I realised that they could bake the same stuff and display perfect results, yet Max's was always dodgy.

Thing is, it seems like Max's viewport calculates tangents almost identically to Maya's baker and viewport. So at least that seems like a more consistent calculation.
Is there any reason why the calculations should be different for max's renderer?!

You're also adding 5000 hard edges to your models to get a correct result in max, you could take those same models with totally smooth normals in maya and bake them with results that are *near flawless*.

EQ you use Modo. How much easier is it to get proper results than in say Max.

You're also adding 5000 hard edges to your models to get a correct result in max, you could take those same models with totally smooth normals in maya and bake them with results that are *near flawless*.

I highly doubt something with many many 90 degree angles would bake well in any app without adding edge bevels or adding in hard edges.

Well I heard of this before, but very interesting to see an example cheers ...
Seems max can only overcome smoothing issues based on raytracer than realtime.

There's a way to get good maps in max (or any other engine) by setup your smoothing groups and uvs properly.
Below is a map I've done with max (realtime viewport) a while ago. (before I switched to xnormal)

Anyway thanks a lot MoP, I was using xNormal without even knowing I could get away with less smoothing errors.
(I have to revise my normalmap article now)

I highly doubt something with many many 90 degree angles would bake well in any app without adding edge bevels or adding in hard edges.

So, you're saying you use NO hard edges, and get perfect results in max?....

Clearly you can't be saying that, or you would be showing examples that prove everyone in this thread wrong.

This is a pretty good example of what happens in max, even using hard edges with a pretty clean mesh, no outrageous issues or anything, you get visible smoothing errors, in any other app(maya, xn) these areas should render perfectly without hard edges, but in max you get errors even with them!

Thing is, it seems like Max's viewport calculates tangents almost identically to Maya's baker and viewport. So at least that seems like a more consistent calculation.
Is there any reason why the calculations should be different for max's renderer?!

some years ago I looked into that stuff when doing the exporters for luxinia, which uses the nvmeshmender to create tangentspace (I could find references to it in the maya sdk code, too).

why should the calcs be different for max renderer?

Quality! in the scanline renderer you could use a more costly interpolation than in real-time (with connectivity... could make some nice smooth normals and therefore smoother tangentspace basis too) I speculate so, since that very part is not exposed in 3dsmax sdk. They do have some sample code how they create their tangent space so.

the viewport rendering is mostly a "preview" for their renderings, I don't think that the "game ready" stuff really has a big lobby with them. Else they could have been much nicer and expose stuff to do custom tangent-space and modify baker and so on, which is common in games as you know.

Why no standard? Well it's all about optimization. You can feed the tangent space with 3 vectors, or with 2 and assume the last one is perpendicular (just need the sign for direction, ie. vector3 for normal and vector4 for tangent and sign encoded in .w) One could per-pixel normalize all the tangentspace vectors again or not... so yeah even for this simple stuff plenty options... it's a ugly situation indeed. But frankly I think not that many variants are around there anymore. Also keep in mind that say crytek had their own baking tools as they started when baking in max or maya was poor still, or to make sure that nomatter where stuff came from run through own tools... then when everyone uses their own tool, they also come up with their own ways, as there is no need to ask "other devlopers" or whatnot hehe

So, you're saying you use NO hard edges, and get perfect results in max?....

Clearly you can't be saying that, or you would be showing examples that prove everyone in this thread wrong.

No, I'm saying that you CAN'T make a model with tons of 90 degree angles, then simply put soft edges on all of it and bake it perfectly in maya or xnormal, which seemed to be what you were implying. When you do that, you always get those cool warping errors in realtime apps.

As for that example, yeah I've seen that happen on some of my test bakes before. If you triangulate the low poly mesh it doesn't seem to happen at all.

CrazyButcher: Ahh, makes sense, I forgot that some methods were optimised for performance and therefore slightly different. Still, it's annoying that there's no option to change the method for either renderer or viewport, since it means inconsistency internally in the app, which I find quite crazy.

No, I'm saying that you CAN'T make a model with tons of 90 degree angles, then simply put soft edges on all of it and bake it perfectly in maya or xnormal. When you do that, you always get those cool warping errors in realtime apps.

But that is the thing, in maya you can get near flawless results, even when you have what most people would consider "bad" smoothing, much much better than what you get from max. This trickles down to fix virtually every "slight" smoothing error you would ever have, not just the big stuff.

The same can be said for XN, if you bake and preview in XN, but not as accurate as maya.

As for that example, yeah I've seen that happen on some of my test bakes before. If you triangulate the low poly mesh it doesn't seem to happen at all.

I've tried triangualting the mesh in max, and you can sometimes get better results by flipping the invisible edges too, but there are still obvious problems with max's renderer.

At the end of the day, there are tweaks you can make, hard edges, you can load up extra bevels to improve the quality, but in max you're just smoothing over the symptoms, and there are serious problems underneath.

I've used Epic's SHTools plugin for Max to render out normal maps for Unreal in the past. The resulting maps were supposedly more "correct" than Scanline or Mental Ray bakes. So I'm guessing that's how they handled the odd baking properties of Max.

MoP, not so sensational for me, I've known this for over a year, I even mentioned it a few times here on PC, but nobody seemed to care

You can see why it's doing it, by comparing a render of a sphere or any curved object with how it looks in viewport. You'll see the shading looks more correct when rendered with scanline, because it does some sort of interpolating correction, and not just linearly blending between vertex normals.

The crazy thing is, that Max' built in "Best" or SM3.0 default viewport material has these corrected normals in realtime. This is a shader that was added since Max 2008 and is embedded into 3DS Max from 2009 onwards (in 2008 you can actually look at the files).

Here's a screen that illustrates things:

Left: SM 3.0 realtime "improved" normals with default max material display
Center: the regular way any shader or games displays this mesh
Right: tangent space normalmap applied, you can see the incorrect baking Max has done, unable to fix the smoothing at the back.

All of these are the exact same mesh.

Now that there's finally some attention for this issue, I'd like to expand it to trying to find out what the calculations are to correct the normals like that. I tried looking for it but got nowhere...

Also, Xnormal is being mentioned as the alternative, but I have to say that Max's supersamplers are FAR superior to Xnormal. If you turn the global map sampler to Hammersley at 1.0, you get extremely nice anti-aliasing on sharp edges in the map, something where Xnormal fails. Santiago told me months ago he planned to add supersampling to Xnormal, but I'm still waiting for it. Until then, Xnormal is not an option, the maps from Max do look better, regardless of the incorrect tangent basis.

If you want better AA in xn, you can just render at 2x, which is essentially the same thing as "supersampling" in max isnt it? It will take 4x as long, but for normals that shouldn't be an issue(turn it back down for ao as it isnt quite as important).

I've had max 2010 display wierd shit when my normals are applied in realtime using the DX display. But, that was usually more so having it put seams where seams didn't even exist.

At work I really never go off of what it shows in max as far as most anything is concerned as far as textures go.

But what I have learned , is that if you apply your Normal Map and display it in realtime with the DX display, and then apply an edit normals modifier, and collapse the stack with it still applied, it fixes up your LP models normals to work better with the normal map normals, fixing weird issues.

Also, as far as hard edges and normal map bakes go, Yes, they do fuck up your results usually, but it depends.

I've basically learned to do an autosmooth by 80 degrees on the entire model before baking, which usually works fine. But sometimes you still need to separate certain faces to different smoothing groups to fix issues.

Also, Xnormal is being mentioned as the alternative, but I have to say that Max's supersamplers are FAR superior to Xnormal. If you turn the global map sampler to Hammersley at 1.0, you get extremely nice anti-aliasing on sharp edges in the map, something where Xnormal fails. Santiago told me months ago he planned to add supersampling to Xnormal, but I'm still waiting for it. Until then, Xnormal is not an option, the maps from Max do look better, regardless of the incorrect tangent basis.

no, you don't want to have to load an external app everytime you do a test bake that's just plain counterproductive. this is incredible that the only workaround is "don't use it" normal maps have been out for several years now this is just plain stupid that there's not some kind of standard that every app accepted to share, and we, as users pay for this in terms of efficiency...

oh and you'd be surprised how many clients ask for max/whatever app viewport screencap and don't want to hear about marmoset , etc simply because the app is what they use, not marmoset or xnormal or whatever.

I agree Autodesk should fix max's shittiness for displaying normals, among other things, but I won't hold my breath.

But, usually for my bakes, I just look for errors, and weird gradients, and if it looks clean, I roll with it and move on, and usually it works out fine for me. I don't test in max, because our pipeline makes it fast to see it in-game, which is what counts anyways.

oh and you'd be surprised how many clients ask for max/whatever app viewport screencap and don't want to hear about marmoset , etc simply because the app is what they use, not marmoset or xnormal or whatever.

And then they'll have to export from max to their engine and it will probably be different again. I guess if you're a freelancer you wouldn't care about that step.

Most engines I've used have slightly different normal calculations to what you'll see in max or maya. The lighting can look very different and the shaders may be more or less complicated. Obsessing over minor differences between max and maya doesn't seem as important to me when dealing with the gap between your engine and your modeling tool.

And here we come to the conclusion - better have your engine be able to bake the maps by itself.

You know, just like the very first released game to utilize normalmaps - Doom3!
Its renderbumb output is different from both Max's and Maya's, and deals with extreme smoothing much better if I understand and remember correctly.

Regarding Epic and MAx bakes, at least for UT3 : the games shows some strong cases of normalmap mirroring issues (the red trainwreck in the first level), making me think that at that time, they indeed baked stuff and moved along if the result was "good enough" (especially since its mostly rusty and gritty materials). Gears looked cleaner tho so who knows. Also maybe their tangents were stripped on import - just like the vertex normals being recalculated. Making organic segmented meshes a royal pain to do...

Tangents info is important for curved shapes and seams ... maybe the Max baker just doesnt even consider them at all?

And yeah I wouldn't be surprised to hear that the viewport/scanline differences simply comes from the teams behind these two tools never having met each other. :P

Also I personally gave up on trying to preserve quads on work meshes before or after the bake. There is just way too many factors possibly going wrong at any time in the pipeline. Tris everywhere, I'd say.

So if I'm using xNormal's SBM exporter in Max, and I check "export tangent basis", does that mean that whatever tangent basis weirdness max has is going to carry over to my xN render? Or is that like a totally different thing?

You can get perfectly good results in max.. if you're willing to significantly increase the polycount of your mesh, which isn't ideal. In my experience though, MoP, it's not that max produces bad results, but that maya produces better results than pretty much any other major pipeline out there.

At 3Point we've discussed building bake and viewport support for maya style normal maps in 3ds, though it's unknown to what degree that would be adapted by development houses. The idea is we'd like to produce the best assets possible for clients, and this is very much tied into the baking process.

The crazy thing is, that Max' built in "Best" or SM3.0 default viewport material has these corrected normals in realtime. This is a shader that was added since Max 2008 and is embedded into 3DS Max from 2009 onwards (in 2008 you can actually look at the files).

Wow, I didn't know about this, where is the setting to turn this on?

I only recently upgraded to a graphics card that supports SM3.0 (was using a GeForce 7600 before a couple of weeks ago) so I wouldn't even have had that option

Pior's got a point though, there are only two ways to ensure that your normal-maps look correct in-game (of which he mentions one):

Bake your normal-maps in your game engine or custom tools set up by the programming team so that you can guarantee perfect results.

Make sure your game engine is set up to read normals from your 3d application, and derive tangent basis in exactly the same way.

There is also option 3, which is to do both of the above things, which is what we do at SD

Eld: I think you're worrying a bit too much about triangulation here - the point of the normal-map is you should see no triangulation at all once you have baked. If you still can, then it's because either your normal map display is wrong (as shown on the middle model in Xoliul's screenshot), or your mesh is constructed in such a way that makes it impossible for the normal-map to compensate correctly (eg. having a smooth edge around a greater than 90 degree corner).
I can see your point though, you can see the triangulation direction in that screenshot, but if the normal-map was being used correctly (as on the far left) then it should not be an issue.