Texture state usage warning

If you’ve arrived at this post by Google searching for the “base level
inconsistent” error message, welcome! Jump down to the ‘Solution‘
section below, to skip the preamble and get right to the possible fixes.

MMORPG Tycoon 2 is being built inside my game engine, VectorStorm,
rather than inside an off-the-shelf game engine. This is in part for historical
reasons (the original MMORPG Tycoon was also implemented in this engine, and I
wanted to re-use a lot of the simulation code), and also because at the time I
started work on MMORPG Tycoon 2, the other (more mainstream) engines like Unity
didn’t yet support some of the rendering features which I needed to make the
game viable.

The great thing about writing your own engine is that you get complete, easy
control of the full rendering pipeline. The awful thing is that sometimes you
have to deal with low-level problems which aren’t entirely clear.

This post is about one of those times.

This post is about yesterday.

"Actor Test" mode

Yesterday, I was starting work on a minor change that I’ve been planning for
MMORPG Tycoon 2 forever, involving the way that players die and respawn. I
wanted to temporarily leave a gravestone behind where a player died. I have
several gravestone models to use for this purpose, but I wanted to check them
for size, relative to the players.

Since the game takes a short time to load into a real game (about five seconds
on my computer), I decided to check the gravestones in a smaller, quicker testbed.

Specifically, I was running in “Actor Test” mode (visible above), since this
mode already displays an in-game character; all I needed to add was the gravestones,
and I could quickly tell if they were of a reasonable size, or if they needed to
be adjusted.

Actor Test mode is particularly nice because it launches pretty much instantly;
there’s no world to load, no AI, just a single character model (here, the
dragon which was visible in the greenlight trailer). That’s five whole
seconds saved!

Now, I need to mention that when you’re rendering using OpenGL version 3 or
later, you can set up your OpenGL as a “debug context”, in which the driver
sends you back informational messages about what’s going on. If you have an
error in your OpenGL commands, the driver will send you a message to tell you
specifically what you did wrong. Often, the driver can also warn you about
performance pitfalls, if you’re doing something silly.

I always run MT2 in an OpenGL “debug” configuration when I’m writing code. It’s
fantastically useful. Or at least, it usually is. But in this case, what the
driver told me was this:

So.. yeah. That probably means about as much to you as it did to me. Doing a
web search didn’t help, either; it seems like nobody understands what this
message really means.

Ordinarily, you get this kind of warning message if you’re drawing from a
texture that has incorrectly sized mipmaps, or not enough mipmaps, or.. maybe
other problems. And the number tells you which texture unit has the problem;
in OpenGL, textures inside a draw are numbered, and this message says that
Texture 0 is the one with a problem.

But.. if you look at the dragon in the image above, you’ll notice that it
doesn’t actually have any textures. It’s entirely flat-shaded. There is no
Texture 0 to have a problem, so the message doesn’t make much sense.

Now, it is reading from a shadow map texture to figure out which bits of the
model are in shadow. But the shadow map isn’t using texture unit 0; it’s
texture unit 6! And I’ve verified that the shadow map is completely fine;
we’re not actually failing to read from the texture at all. In fact,
everything seems to be working 100% correctly, apart from that persistent
warning message, which it’s spitting out several dozens of times per rendered
frame.

And so, in an effort to save those five seconds of time spent loading into the
game, I began a debugging effort that consumed most of the day, trying to make
‘Actor Test’ mode (which won’t even ship with the game) render without
generating warnings.

I never claimed to be smart.

Solution

The most common cause for this warning is simply forgetting to bind your
texture onto the correct texture unit before drawing. I’ve actually hit this
one a couple of times in the past; if your shader wants to use a texture from
unit 0, and you haven’t actually bound a texture onto unit 0 at all, then
you’ll get this warning.

This is pretty easy to spot by capturing a frame in RenderDoc and
looking at what texture units are set to for the draw call, or otherwise just
checking to see whether the texture is actually giving you correct texture
values.

If the message goes away, it means you need to fix your mipmaps! (Or leave
them disabled.)

Those are both real issues, and are both common. But neither is what was
happening to me, yesterday. No, my thing was much more subtle.

In VectorStorm, we use a “material” system which lets the user define various
rendering parameters. Amongst these are up to 16 “slots” which can contain
textures which will be automatically passed to shaders. Those 16 texture maps
get mapped to Texture units 0 through 15.

In the case of the dragon shown above, we used just one texture; the “shadow
map” texture, which I’ve defined as always sitting in the material’s texture
slot #6 (and therefore, texture unit 6 while rendering). This is convenient,
because it means I can share code between shaders; everything can check for
shadows just by looking in texture unit 6.

Inside the GLSL shader, this setup looks like this:

#version 330
uniform sampler2D textures[8];

textures[0] is a sampler pointing at texture unit 0, textures[1] is
pointing at texture unit 1, and so on; VectorStorm just sets that up
automatically.

In the case of the dragon model shown above, the shader only ever accessed
textures[6]. The only purpose of the other sampler uniforms in the array was
to fill space, so that textures[6] would automatically be pointing at texture
unit 6, which contained the texture from the material’s texture slot 6.

Here’s what was going on: it seems that on modern NVidia drivers, if you
have an array of texture sampler uniforms in your shader, and you access any
element of the array, it only checks the very first sampler in the array for a
valid texture, and it checks that first sampler even if that’s not the one you
sampled from.

So.. it actually seems like this specific instance of the message was caused
by a benign driver bug. It doesn’t indicate any real problems.

…but if you’re like me, it bothers you anyway, and you want to fix things so the
message actually goes away; you don’t just want to ignore it.

What I did to make the warning go away was to just not set up samplers in an
array after all; instead, explicitly declare just the samplers you actually need:

Now the ‘shadowTexture’ sampler points to texture unit 6, and the driver checks
the correct texture for validity, while rendering.

It’s a bit more work than the automatic “texture slots” system using an array
of uniforms, but it means that every sampler defined in the shader actually
gets used. And it seems to keep NVidia’s drivers happy. So.. yay? It’s
probably better code, anyway.

Ironically, we never got this error message in the full game, even when drawing
this dragon model. That’s because VectorStorm never cleared textures out of
texture units after using them.

While this dragon only uses texture unit 6, and has nothing in any of the other
texture units, something I had drawn previously would have put something into
texture unit 0, and the engine would just leave it there afterward until somebody
else needed something different in texture unit 0.

Leaving somebody else’s texture in unit 0 wouldn’t do any harm, since the
dragon rendering wasn’t going to try to read from it, so why bother spending
any time clearing it out again, right?

And since there was a valid texture in texture unit 0 (even though the shader
wasn’t actually going to read from it), the NVidia driver was happy, and gave
no warnings in the real game. It wasn’t until I moved to this cut-down ‘Actor
Test’ mode where I wasn’t drawing anything before the dragon, that I bumped
into the issue at all.

I hope this helps somebody else who receives this warning message! Took a
rather shocking amount of time to track down where the message was actually
coming from; if I can save somebody else that time, that’d be awesome!

Next time, I promise I’ll talk about the game again. Technical posts are fun,
but.. quite possibly only for me. :D