Tutorial - Closing the Open-Manifold Mesh (3DS Max)

This is a simple tutorial that goes over the problem that sometimes arises when making a model for a game such as FS2. To start off, we'll go over a few definitions so that you'll know some terminology:

What is a Manifold Mesh?A manifold mesh is essentially just a normal 3D model that is mesh or poly based. To my knowledge, patchs and NURBS based objects are not considered as a manifold mesh.

There are three types of manifold meshes that should be known: closed-, open-, and non-manifold meshes.

What is a Closed-Manifold Mesh?A closed-manifold mesh has all of its edges sided by exactly 2 faces.

What is an Open-Manifold Mesh?An open-manifold meshhas at least 1 edge that has only 1 face attached.

What is a Non-Manifold Mesh?A non-manifold mesh has at least 1 edge that has more than 2 faces attached.

Open- and Non-manifold meshes can have a significantly less Tri and Vertix count than closed meshes, but cannot be used in games such as FS2. This is because FS2 uses the mesh for collision detection of craft and weapon ordinance, as well as calculating the objects "mass." This latter calculation absolutely requires closed-meshes... and may be imagined as filling the inside of the mesh with "mass" just like you would fill up a bucket with water.

Luckily, there is a procedure available for 3DS Max users that allows the modeler to stitch up and close the open mesh.

Finding the Open- and Non-Manifold Errors

Select your object and convert it into an Editable Poly.

Right-click in one of the veiwports to open up the Quad-Menu, and then select "Convert to Editable Poly"

Convert to Editable Poly

Apply the STL Modifier.

Left-click the "Modifier List" drop-down box located on the right-hand tool pane to open up the modifier list. Scroll down and search for "STL_Check" in the listing.

Note: If it does not appear, or you cannot find it, you may have to update your version of 3DS Max or look for a plugin online.

Apply the STL Check Modifier

Setup the Modifier Parameters.

On the right-hand tool pane, the "Parameters" rollout for the STL Check modifier should be seen.

Edges that have some sort of open- or non-manifold mesh quality to them will be highlighted.

Types of Errors and How to Fix Them

There are four types of errors that the STL Check can find:

Open Edge

These are edges that have only 1 face attached to them. Most common error.

Open Edges are typically solved by selecting "Border" on the Editable Poly and then "capping" them.

Cap the Gap

There are sometimes, however, edges that are flagged as errors when polygons skip one or more vertices on one edge. These are nearly impossible to spot visually, and can only be found by using the STL Check modifier.

To solve these errors, you must select the polygon that common to the entire highlighted edge or edges, delete it, and then cap the gap.

Red Highlighted edges are "open," as determined by the STL Check

The front polygon has been deleted, exposing a new border

Cap the Gap

Success! The top edge is no longer an error

Double Faces

These are faces that "overlap" one or more other faces.

During the conversion to Editable Poly, 3DS Max tries to get rid of the double faces (or Tri's) when it re-triangulates the Polygons as set by the Editable Mesh edge visibility. If the double-faces have at least one invisible edge, then they will be merged into a polygon.

If faces still overlap, then the modeler must delete one or more of the overlapping polygons and cap the resulting border created by the deleted polygons.

Spikes

These are faces that only share 1 edge with the main object.

By default, Editable Poly's automatically separate spike's into a separate "element." therefore, you will most likely pick up on Spike errors as "Open Edge" errors instead.

If the modeler wants to keep the spikes, then they will have to experiment to find a good solution, such as replacing the spike with a pyramid.

Multiple Edges

These are edges that "overlap" one or more other edges.

They more than often occur in Editable Mesh objects, but are virtually eliminated by Editable Poly objects.

ConclusionThe Editable Poly type object is an insanely useful tool for gaming, especially when dealing with objects that have already been UVW mapped. They are also very useful in optimization, and, as this tutorial showed, invaluable in repairing non- and open-manifold meshes.

Remember to check your UVW mapping once you have done your modifications. There are some times when the UVW is changed during your operations.

Addendum: Reducing the Rendering Load Caused by the Addition of Unseen Faces

In many instances, there will be faces that will never be seen by the players on closed-manifold meshes. This poses an interesting problem efficiency wise, because the faces will still have a texture applied to them.

Luckily, for FS2 there is a special default texture known as "invisible.tga"

This texture is a unique stock texture that is 10x10 and has an alpha value of 255, but most importantly it has a significantly lower memory usage.

The solution then should become readily apparent: simply select all faces that will not be seen by the player, and apply the invisible.tga texture to them.

As a modeling aide in 3DS, you can "hide" the diffuse map (invisible.tga) in the maps rollout of the material editor, and use the "Self Illumination" and "Opacity" settings to make the invisible areas visible. Don't worry, when you export your model, the map will still be applied to the faces.

Additionally, there are some faces drawn between very close together points, such as a scaled down cap on a cylinder to represent a point. The STL Check modifier will not detect these. You can, however, easily weld together any vertix that's within a specified distance to another vertix by first selecting all verticies and then performing a "Weld within threshold" command.

Example of a 0m2 surface. The cone on the right is really a cylinder with the top face scaled way down.

Select all verticies of the object in question, and then press the "Settings" button near the "Weld" command either in the quad-menu (right click) or on the sidebar.

Finally, press the up arrow (highlighted) until the "After" vertix count is lower than the "Before."

You can rotate and pan around in your veiwport while this dialog is open, which is useful to check for any possible distortion that might have occurred. If there is distortion, then you must either reduce your threshold or reduce your vertix selection to only the corners.

Re: Tutorial - Closing the Open-Manifold Mesh (3DS Max)

Addendum: Reducing the Rendering Load Caused by the Addition of Unseen Faces

In many instances, there will be faces that will never be seen by the players on closed-manifold meshes. This poses an interesting problem efficiency wise, because the faces will still have a texture applied to them.

Luckily, for FS2 there is a special default texture known as "invisible.tga"

This texture is a unique stock texture that is 10x10 and has an alpha value of 255, but most importantly it has a significantly lower memory usage.

The solution then should become readily apparent: simply select all faces that will not be seen by the player, and apply the invisible.tga texture to them.

As a modeling aide in 3DS, you can "hide" the diffuse map (invisible.tga) in the maps rollout of the material editor, and use the "Self Illumination" and "Opacity" settings to make the invisible areas visible. Don't worry, when you export your model, the map will still be applied to the faces.

Re: Tutorial - Closing the Open-Manifold Mesh (3DS Max)

But doesn't this cause the engine to attempt an additional render pass with the "invisible" texture?I vaguely remember that the engine doesn't render polygons with the "invisible" texture applied, but does it really stop the render pass early enough that you get a performance gain?

"I don't think that people accept the fact that life doesn't make sense. I think it makes people terribly uncomfortable. It seems like religion and myth were invented against that, trying to make sense out of it." - D. Lynch

Re: Tutorial - Closing the Open-Manifold Mesh (3DS Max)

But doesn't this cause the engine to attempt an additional render pass with the "invisible" texture?I vaguely remember that the engine doesn't render polygons with the "invisible" texture applied, but does it really stop the render pass early enough that you get a performance gain?

Well, that, I'm not sure of. PCS2 responded very well when I applied a high-poly object with the texture, but there needs to be in-game testing to see if it's a truly viable option.

Re: Tutorial - Closing the Open-Manifold Mesh (3DS Max)

Just to clarify a few things: FSO works fine with open edges, as does PCS2.

The part that would be causing problems here is the save to DAE from the 3d app - in this case Max. I can only speak from experience with the Blender DAE export script, but it does work fine with open edges. As such, if your DAE save operation is ok with it, then the best solution by far is to delete the internal faces completely, because that way they don't interfere with the rendering system, the collision system or the invisible texture system. (invisible polys do still get counted for collision calculations unless you have the dont-collide-invisible flag set in that ship's table.)

TBH I suspect there's more at work in this case than a problem of manifold vs non-manifold.

About the invisible texture trick - it's a clever idea definitely, however I would only recommend using it if you 1) HAVE to use completely manifold meshes (due to your DAE exporter as described above) and also 2) you have more than about 1000 faces that are completely internal and will never be seen. Otherwise it'd be extra work for both you and the collision detection system that would be basically of no benefit to the final result.

Anyway all other common errors in your tut are very much valid ones modellers should learn how to deal with. Nice work. I'd also suggest adding how to find and deal with 0m² area faces, such as occur if you were making an antenna by taking a cylinder and shrinking one end-face of it down to something absolutely tiny so that it LOOKS like a point, but is not. Does Max have a way to detect that?

Re: Tutorial - Closing the Open-Manifold Mesh (3DS Max)

Just to clarify a few things: FSO works fine with open edges, as does PCS2.

The part that would be causing problems here is the save to DAE from the 3d app - in this case Max. I can only speak from experience with the Blender DAE export script, but it does work fine with open edges. As such, if your DAE save operation is ok with it, then the best solution by far is to delete the internal faces completely, because that way they don't interfere with the rendering system, the collision system or the invisible texture system. (invisible polys do still get counted for collision calculations unless you have the dont-collide-invisible flag set in that ship's table.)

TBH I suspect there's more at work in this case than a problem of manifold vs non-manifold.

I keep hearing about collision problems on some models because they were non-manifold, so the safest bet is to have at least 1 LOD that is a closed-manifold. Non-manifold's are by far more poly efficient in comparison to most manifolds, but can't hold water (theoritically speaking).

Quote

About the invisible texture trick - it's a clever idea definitely, however I would only recommend using it if you 1) HAVE to use completely manifold meshes (due to your DAE exporter as described above) and also 2) you have more than about 1000 faces that are completely internal and will never be seen. Otherwise it'd be extra work for both you and the collision detection system that would be basically of no benefit to the final result.

I plan on doing some collision tests of the mixed LOD's, with detail-0 a closed manifold and the others an non-manifold. I'm hoping that my guesses will work.

Quote

Anyway all other common errors in your tut are very much valid ones modellers should learn how to deal with. Nice work.

Yay, I did something right! Thanks.

Quote

I'd also suggest adding how to find and deal with 0m² area faces, such as occur if you were making an antenna by taking a cylinder and shrinking one end-face of it down to something absolutely tiny so that it LOOKS like a point, but is not. Does Max have a way to detect that?

The STL check will only find what's mentioned in the tut, it doesn't find 0m2 surfaces. However, as BlackWolf mentioned, you can do a "Weld within threshold" command to take care of this easily.

Re: Tutorial - Closing the Open-Manifold Mesh (3DS Max)

Collision problems won't be specifically caused by non-manifold models - they'll be caused by some other issue that may or may not be related to the reason the mesh is non-manifold. IIRC, there were massive collision detection issues on the colossus, which I *think* were completely fixed by triangulating it prior to DAE export for example.

If you look at any model which displays sharp edges (ie, has not been 100% smoothed all over) then those edges you are looking at in PCS2 or FSO are actually holes, meaning pretty much every ship in FS is non-manifold in some way. The way it works is that each vertex on a model has a normal, and that normal is calculated as I think an average direction of the normals of all the faces that vert is a part of. Shading each face involves interpolating between the vertex normals of that face. As such, AFAIK in FS the only way to have a sharp edge where one face does not affect the shading of the one on the other side of the edge is to have separate verts that share the same location but have different normals. You can prove it by comparing the (true) vertex count of a faceted sphere vs a smoothed sphere after conversion. It's far lower on the smoothed sphere because there is no duplication of verts.

Oh and for the 0 area faces - bad example sorry. Try a scenario more like this: http://www.hard-light.net/wiki/index.php/File:BlenderErrorHoles.jpg (Imagine the hole on the bottom being filled with a face so that a STL check does not detect it) Here all 3 verts of the face are in a line rather than being close enough to weld. You'll need some kind of special selection tool to detect that. In blender for example you have a "Clean meshes" script that will delete 0 area faces, thus allowing select-non-manifold to detect those bits as problem areas.

Re: Tutorial - Closing the Open-Manifold Mesh (3DS Max)

I don't know of any built-in tools to detect those, but there might be a Maxscript floating out there that would try to calculate the area of all selected faces, and then clean up the ones that have lower than some threshold...

Re: Tutorial - Closing the Open-Manifold Mesh (3DS Max)

If you look at any model which displays sharp edges (ie, has not been 100% smoothed all over) then those edges you are looking at in PCS2 or FSO are actually holes, meaning pretty much every ship in FS is non-manifold in some way. The way it works is that each vertex on a model has a normal, and that normal is calculated as I think an average direction of the normals of all the faces that vert is a part of. Shading each face involves interpolating between the vertex normals of that face. As such, AFAIK in FS the only way to have a sharp edge where one face does not affect the shading of the one on the other side of the edge is to have separate verts that share the same location but have different normals. You can prove it by comparing the (true) vertex count of a faceted sphere vs a smoothed sphere after conversion. It's far lower on the smoothed sphere because there is no duplication of verts.

I can't be 100% sure whenever PCS2 is involved, but I still dare to say that's not true. I can export a model all the way from .max to .dae to .pof to .dae to .max to .dae to .pof and have the smoothing retained (although it does have a problem; after importing a .dae in Max, I can't actually manipulate the smoothing groups even though they are rendered right) without any kind of silly duplication of vertices in order to simulate smoothing.

I can't recall ever having seen such behaviour in any model (although I haven't looked at retail models much) and I get distinctly different-looking shading in-game depending on how I've smoothed the model in Max.

Re: Tutorial - Closing the Open-Manifold Mesh (3DS Max)

after importing a .dae in Max, I can't actually manipulate the smoothing groups even though they are rendered right) without any kind of silly duplication of vertices in order to simulate smoothing.

I also noticed this when importing a DAE file into max, even tough smooth groups can still be edited, it has zero effect on the model. I suspect it has something to do with the vertex normals, but I can't find a way to edit those in Max.

"I don't think that people accept the fact that life doesn't make sense. I think it makes people terribly uncomfortable. It seems like religion and myth were invented against that, trying to make sense out of it." - D. Lynch

"I don't think that people accept the fact that life doesn't make sense. I think it makes people terribly uncomfortable. It seems like religion and myth were invented against that, trying to make sense out of it." - D. Lynch