Right now i'm working on a game engine for a game i want to make, this is mainly for research and learning purposes.

I have most of the rendering base code done, loading meshes from a custom format aswell as materials and other stuff, however, in the last days i've been doing some research about what space partitioning method should i use for my world geometry.

I've been reading about KD-Trees, Octrees (and it's smaller variants) and BSP Trees, out of the three i guess i will stick with Octrees because it allows both outdoor and indoor efficient culling of geometry (or that's what i understood), the game will be mainly indoor but because of the nature of the project i wanted some generic way to make both types of scenes.

I've got the concept of octrees pretty well and i dont think i will have problems when implementing them (Im a self-taught graphis programmer), however there are some concepts that i dont get a clear view when i think about them, mainly precomputed visibility of nodes and what would be the most efficient way to render each visible node.

For the first i thought of the hardcore way, for each node check which nodes are visible (just like portals), this is kinda heavy for a complex scene but i dont mind to spend some time on precomputing visibility, the other i thought is to do the checking on the fly based on the camera frustum bounding box or the frustum planes themselves.

The other is once you determine which nodes are visible, what would be the most efficient way to render them.

The first approach i thought was to build a list of indices of the visible triangles and send them to the rendering API to be processed, main problem with this is that is kind of CPU intensive i guess since you would have to iterate over all the nodes, grab the indices of the polygons inside them and copy them over the active index buffer.

The second was to make a index buffer for each visible node and render each node manually, main problem with this is the memory overhead (i guess that DX/OGL uses extra memory per each created index buffer besides the memory needed to hold the indices itself) and the increased draw calls, both for the drawing itself and the switching between index buffers.

The third is to precompute a index buffer of the visible polygons for each node, same issue with memory as the second method.

I ask the help and opinion of the experts over here about what would you guys recommend me in order to create the best possible design.

I suggest splitting your static data on node boundaries after you have your visibility graph, since this allows you to associate data with nodes of your tree partition. Then you just draw the static geometry associated with that node if the node is visible. If you have instanced geometry in a node, you can just conservatively make the whole object visible - just be sure to do a scoreboard/mailbox check so that you don't draw it once for each visible node it intersects. You may want to do the same for non-instanced geometry if your subdivision is very fine and results in a lot of splits (since this will result in more draw calls).

For dynamic objects, you can easily (via a hierarchical test) figure out which nodes it intersects, and if it intersects a visible node, then it is treated as visible.

BTW, How do you intend to compute the PVS for the nodes? (which approach/algorithm - it's a difficult problem)

For best results, I suggest a general axially aligned BSP tree, since you ideally want to keep your leaf nodes cubic.

I suggest splitting your static data on node boundaries after you have your visibility graph, since this allows you to associate data with nodes of your tree partition. Then you just draw the static geometry associated with that node if the node is visible. If you have instanced geometry in a node, you can just conservatively make the whole object visible - just be sure to do a scoreboard/mailbox check so that you don't draw it once for each visible node it intersects. You may want to do the same for non-instanced geometry if your subdivision is very fine and results in a lot of splits (since this will result in more draw calls).

For dynamic objects, you can easily (via a hierarchical test) figure out which nodes it intersects, and if it intersects a visible node, then it is treated as visible.

BTW, How do you intend to compute the PVS for the nodes? (which approach/algorithm - it's a difficult problem)

For best results, I suggest a general axially aligned BSP tree, since you ideally want to keep your leaf nodes cubic.

Thank you for the answer, the AA BSP Tree might be a good idea.

I already thought about how to filter dynamic objects just in the same way you said, basically each entity would know in which cell its located so the entity knows what's visible (useful for IA), same as player basically.

My idea to compute the PVS was to go raycast from each cube corner to all the other cubes, main problem with this is that it will be really expensive based on the fact that each cube have 8 childs, i was thinking to just raycast to the "bigger" cubes (either by area or polygon count) this way i would be able to still skip rendering of hidden geometry and not take ages to compute the visibility.

My idea to compute the PVS was to go raycast from each cube corner to all the other cubes, main problem with this is that it will be really expensive based on the fact that each cube have 8 childs, i was thinking to just raycast to the "bigger" cubes (either by area or polygon count) this way i would be able to still skip rendering of hidden geometry and not take ages to compute the visibility.

I think that you will find that you need to ray-cast from a lot more than the corners to prevent popping/false-invisibility. If you subdivide far enough, this won't be an issue, but then you land up with an extremely dense octree that takes ages to compute (most of the computation is redundant since you are mostly sampling the same lines again and again). For best results you will likely need to densely sample the full line space between your source and destination nodes. This can be done by ray-casting from all over the surface of your source node (to all over the surface of your destination node, stopping if visibility is established).

Chapter 4 talks about using the GPU for sampling visibility. The hardware at the time was dated: I expect that today you could avoid the readback by using occlusion queries or compute shader processing of the z-buffer. Chapters 5 and 6 talks about computing the exact visibility set, but the math is a bit hairy, and it is probably overkill for mose use cases.