I could not find good tutorial about octrees so tried to make one on my own.I have a lot of non-moving/static objects in my scene so i thought i could speed up culling with octree instead with camera frustum-bbox check with all of these. Can you spot what am i doing wrong?

Stack overflow means you have too many recursive functions (among other unrelated things).
You need to check the depth at which you stop generating nodes within the octree. Usually 8 nodes deep, including the head node, is the maximum depth for an octree.

A quick glance at your code shows no sign that you ever terminate the creation of the octree. You just keep creating more and more nodes infinitely deep until it overflows the stack. levelCnt is being incremented but never used.
Additionally, it is “used” incorrectly. The depth of the tree must be passed as a parameter to the function that recursively creates nodes.

You will overcome this problem by adding a check for the level of your current node. Quick example (please put it together with the rest of your code yourself):

[source lang="c++"]void create(std::shared_ptr<OcTree> _tree, int level) {... all the rest, test for intersections and stuff ... if(cnt <= 1 || level > max_level) { ... all the stuff you do in your original code, except that you may have a list of nodes instead of only one ... } else { .... create(..., level + 1) ... }}[/source]

Hope this is understandable, don't really have the time to go more in depth at the moment.

It got rid of overflow but i am doing something wrong in my create function (at least i suspect the problem is there) because i got only few objects displayed compared to what i can see with camera. Please help me further if you find time, thanks.

Hi there,
I can't read your code - too much stl :/
But i remember the pitfalls with octrees, so that might be helpful, if you don't know it:

The problem is: What to do with an object that is split by one or more grid planes?
Put it in each node and draw it multiple times?
Put it in one node and missing to draw it whaen the node is clipped? (guessing thats your problem)

Solution: Put each object to the node, where this condition is true:
objects bounding box center is inside node
AND the largest dimenson from the objects bounding box <= the dimenson of the node.
AND the largest dimenson from the objects bounding box > half the dimenson of the node.

... this results in large objects at the top levels, and small objects at the bottom levels of the tree.

While clipping extrude the nodes by one half of its dimension at each side (so it has 4 times its volume).
This is the volume that surely covers all objects.

Ok, but with your design you'd get the same or better performace with a simpler grid approach, because most objects will be in the small max level nodes.
The result looks more like a sparse grid, that you could clip with large cubes containing smaller cubes without the need of an octree... where's the win from the octree?
If if you use dynamic objects in it, it's slow - too much recursion, shared pointers and dynamic memory allocation... slow.

However, i can't see a bug in the code. Are you sure the frustum check is right?

I still believe its a geometirc issue.
Are the missing objects popping depending on camera or do the never appear?
Did you check after the creation, if each object ist at least in one node?
Did you visualize the node extends, maybe one of the 8 cases has a bug?

For occlusion query it's good to draw objects in raw front to back order, even without occ. test that saves pixel shader due to early z-test.
To achieve that you draw objects by traversing the octree bottom down drawing the near child first.
Here the loose octree has advantage too, because front back order is more precise and you draw large objects (good occluders) first.
but its easy to try that after you found the bug...

Some other things you may or may not end up with later, just to keep in mind:
Node extends and position can be calculated dynamically while traversal, no need to store them for each node.
If you align the grid at integer coords, you can use brachless integer bitmath for very fast searching in the tree.
Mostly it's a win to either point to an array of 8 children (some of them may end up unused), or there are no children.

Personally i use it for the same purpose, but i have not made any of those optimisations listed above
But i did it with a software-renderer that used quadtreecompressed lightmaps, so each scanline was clipping a line into the quadtree,
and there i saw a great speedup.

This adds the object to a vector, right? It looks like it replaces the current pointer with a new one, storing only one object at the end.I'm noob with stl and c++11.

The image looks nice (min / max is good way to define axis aligned boxes),but you need to visualize that inside your engine, to proof that its right.

I'd do this: Turn off Z-Test, traverse the entire tree, if a leaf is found, draw its slightly shrinked extends with a transparent color.Build the color from the nodes memory adress, so you can see different nodes better.For each object in the node, draw a line from node center to object center.

Afterwards draw all objects in wireframe. Navigate through the scene, and check if the objects are linked correctly to the nodes.If an object has no line, or the line points to a node far away, you know more.

If still anything looks right, visualize the cameras viewing pyramid (to see that, you need two cameras, one to observe and one for yourself to fly around).While traversing octree, draw nodes extends in white wireframe, if they fail the observeds cameras frustum check.

...that's some work, but you will surely see the bug - from another perspective - and then its easy to find it.And its good for future bugs.