Quick Peek: Ambient Occlusion

We use vertex coloring for all our meshes, and this presents a bit of a problem when rendering ambient occlusion since it’s very easy to see the interpolation artifacts. Another issue which adds a kink to dealing with AO is that we use a faceted ‘low-poly’ look. We generate the face normals on at run-time so that we can have just one vertex in each position (rather than one per face). Generating face normals in the geometry shader saves us both space and time since the transform cache gets better utilized.

No ambient occlusion

If we just have one vertex at each position, then it’s not obvious in which direction the AO normal should point to get smooth occlusion. Most modeling packages will split the normals and AO based on a certain smoothing or ‘hard edge’ value. Even if forcing all edges to be smooth, it requires some hand tweaking to get the AO to look ok. Maya LT in particular is a nightmare to get nice looking AO with.

I took half an hour over lunch today to test out generating AO inside our mesh compiler (which transforms the mesh from FBX into the game format). It’s relatively easy to test out since it just requires a ray vs line segment intersection test which we already have for our physics engine. You just fire a bunch of rays from each vertex (or a small offset from it) in various directions up to a maximum distance, then take the proportion of misses to rays-cast as the final occlusion value.

Doing it in the mesh compiler isn’t ideal since it may have to run even when the geometry doesn’t change, but it is consistent and doesn’t require any complicated steps in Maya.

Since we can’t rely on using a particular normal direction, I wanted to try just casting rays around the entire sphere from each vertex. Compared to casting over a hemisphere from a face, this is obviously more work for each vertex, but because we don’t split vertices it’s actually a quicker overall. By doing it this way, some rays will actually be cast inside the mesh, but I’m doing double-sided intersection tests so this isn’t an issue if the mesh is closed.

Triangle intersection with 64 rays per vertex

The prototype worked fairly well, but was, as expected, very slow (30 seconds for a normal mesh). So I put another couple of hours into it to see if I could get it to a usable state.

By switching to using an AABB tree (also something used by the physics system) to store the triangles, the AO calculation dropped to about 10 seconds or so. Not bad, but this was still only using 64 samples per vertex. As you can see in the image above, it’s a bit blotchy.

Since AO is an approximation anyway, I tried approximating the triangle mesh with surfels (oriented discs). I just placed one surfel at each triangle centroid, with an area the same size as the triangle’s area. It’s less accurate (and full of holes!), but it’s quicker to ray cast against. I could probably work on the positioning of the surfels a bit, since you can see that they can easily get bunched up.

Surfel display (four-vertices per disc)

The surfel approximation causes rays to miss when they should hit, and vice versa. They don’t quite cancel out due to surfels overlapping each other, but I didn’t mind this. The resulting AO is a bit brighter, but this can be adjusted using a curve to bias it back down.

I was able to raise the number of rays cast per vertex to 256, which smoothed out much of the blotchiness. As a bonus, the rendering time came down to 5.5 seconds. It’s not fantastic, but I’ll take it for four hours work (and never having to mess around with baking AO in Maya again).