I'm still struggling with migrating from old OSG 3.2.1 to the latest developer release, now 3.5.8.

I stumbled upon a rather serious performance issue here. When loading an OpenFlight model with a large number of polygons, the optimization carried out in the OpenFlight plugin is unacceptably slow in 3.5.8. It seems the culprit is MergeGeometryVisitor.

Did you try OSG-3.4.x? I'm wondering about when the regression in
performance might have happened.

I have downloaded the file and will have a look.

Robert.

On 17 November 2017 at 16:14, Andreas Ekstrand
<> wrote:

Quote:

Hi,

I'm still struggling with migrating from old OSG 3.2.1 to the latest
developer release, now 3.5.8.

I stumbled upon a rather serious performance issue here. When loading an
OpenFlight model with a large number of polygons, the optimization carried
out in the OpenFlight plugin is unacceptably slow in 3.5.8. It seems the
culprit is MergeGeometryVisitor.

Yes, I tried it now and OSG 3.4 loads the model in the same time as 3.2.1 did. I found a few versions lying around here and concluded that something must have happened with the optimization between 3.5.1 where it works as before and 3.5.6 where it stalls.

I'll keep digging but please let me know if you find something out.

Regards,
Andreas

On 2017-11-17 17:35, Robert Osfield wrote:

Quote:

Quote:

Hi Andreas,

Did you try OSG-3.4.x? I'm wondering about when the regression in
performance might have happened.

I have downloaded the file and will have a look.

Robert.

On 17 November 2017 at 16:14, Andreas Ekstrand
<> (

Only registered users can see emails on this board!Get registred or enter the forums!

) wrote:

Quote:

Hi,

I'm still struggling with migrating from old OSG 3.2.1 to the latest
developer release, now 3.5.8.

I stumbled upon a rather serious performance issue here. When loading an
OpenFlight model with a large number of polygons, the optimization carried
out in the OpenFlight plugin is unacceptably slow in 3.5.8. It seems the
culprit is MergeGeometryVisitor.

One curious thing I noticed is that when I enabled verbose debug
output there was lots of buffer objects being created and destroyed
during the optimisation step. osg::Drawable now assigns
VetextArrayState and VBO's by default for osg::Geometry - this is
required to make VAO support possible. The large number of these
operations suggest lots of creation and merging of osg::Geometry for
this dataset. There is chance tthat this change alone may be causing
a performance slow down.

However, I don't think creation/deletion of osg::Geometry is the crux
of the problem, and may not even be the optimization step - it could
well be down to the nature of the scene graph being created by the
OpenFlight plugin for this dataset. I suspect the source data is
stored in a way that is really inefficient to handle for real-time.
It might be that the OpenFlight plugin just isn't handling the data
well.

As a general rule, running the Optimizer is a last resort for fixing
really bad datasets, ideally datasets should be created in a form that
is appropriate for decent performance right from the start.

Yes, the model is ineffective in the sense that it has 150 000 separate triangles on the same level in the scene graph, that's the nature of basic usage of OpenFlight and I guess that's why the plugin applies an optimization of its own. But this could be optimized and merged much faster before and I'm just looking for a way to get it to behave like that again, to keep my software from being slower in a new version.

Some results from VTune, where I have manually filtered and presented the top two culprits in 3.5.1 compared to 3.5.8 (where I gave up after 2 minutes):

The MergeGeometryVisitor was applied in 3.5.1 as well but didn't take more than about 0.06s. I guess the 3.5.1 results are generally less interesting but at least they show that MergeGeometryVisitor was not a problem there.

I don't know if this gives you some ideas directly or if we need to keep digging. I will have another look shortly either way, but of course hoping for you or someone else to come up with some bright ideas to shorten the time to fixing this problem.

Regards,
Andreas

On 2017-11-19 16:28, Robert Osfield wrote:

Quote:

Quote:

HI Andreas,

I haven't had a chance to dig further.

One curious thing I noticed is that when I enabled verbose debug
output there was lots of buffer objects being created and destroyed
during the optimisation step. osg::Drawable now assigns
VetextArrayState and VBO's by default for osg::Geometry - this is
required to make VAO support possible. The large number of these
operations suggest lots of creation and merging of osg::Geometry for
this dataset. There is chance tthat this change alone may be causing
a performance slow down.

However, I don't think creation/deletion of osg::Geometry is the crux
of the problem, and may not even be the optimization step - it could
well be down to the nature of the scene graph being created by the
OpenFlight plugin for this dataset. I suspect the source data is
stored in a way that is really inefficient to handle for real-time.
It might be that the OpenFlight plugin just isn't handling the data
well.

As a general rule, running the Optimizer is a last resort for fixing
really bad datasets, ideally datasets should be created in a form that
is appropriate for decent performance right from the start.

Back in February, the MergeGeometryVisitor was changed to be applied on Group instead of Geode by Jannik Heller (in CC). This has resulted in substantially longer optimization time of my loaded OpenFlight files with a large number of polygons. I assume it's a general performance problem with many geodes/geometries.

The removeChild call from MergeGeometryVisitor ::mergeGroup is the main culprit and takes about 87% of the time:
group.removeChild(*ditr);

Does Robert or Jannik have any idea about this? I don't feel comfortable enough with the code to make any changes, but this prevents me from upgrading my software to the latest OSG version, while I can't go back either since I need other fixes in 3.5.8. So I'm a bit stuck and would appreciate any help!

Regards,
Andreas

On 2017-11-19 17:13, Andreas Ekstrand wrote:

Quote:

Hi Robert,

Yes, the model is ineffective in the sense that it has 150 000 separate triangles on the same level in the scene graph, that's the nature of basic usage of OpenFlight and I guess that's why the plugin applies an optimization of its own. But this could be optimized and merged much faster before and I'm just looking for a way to get it to behave like that again, to keep my software from being slower in a new version.

Some results from VTune, where I have manually filtered and presented the top two culprits in 3.5.1 compared to 3.5.8 (where I gave up after 2 minutes):

The MergeGeometryVisitor was applied in 3.5.1 as well but didn't take more than about 0.06s. I guess the 3.5.1 results are generally less interesting but at least they show that MergeGeometryVisitor was not a problem there.

I don't know if this gives you some ideas directly or if we need to keep digging. I will have another look shortly either way, but of course hoping for you or someone else to come up with some bright ideas to shorten the time to fixing this problem.

Regards,
Andreas

On 2017-11-19 16:28, Robert Osfield wrote:

Quote:

Quote:

HI Andreas,

I haven't had a chance to dig further.

One curious thing I noticed is that when I enabled verbose debug
output there was lots of buffer objects being created and destroyed
during the optimisation step. osg::Drawable now assigns
VetextArrayState and VBO's by default for osg::Geometry - this is
required to make VAO support possible. The large number of these
operations suggest lots of creation and merging of osg::Geometry for
this dataset. There is chance tthat this change alone may be causing
a performance slow down.

However, I don't think creation/deletion of osg::Geometry is the crux
of the problem, and may not even be the optimization step - it could
well be down to the nature of the scene graph being created by the
OpenFlight plugin for this dataset. I suspect the source data is
stored in a way that is really inefficient to handle for real-time.
It might be that the OpenFlight plugin just isn't handling the data
well.

As a general rule, running the Optimizer is a last resort for fixing
really bad datasets, ideally datasets should be created in a form that
is appropriate for decent performance right from the start.

Forgot to mention that the geometry before and after optimization is identical in the different OSG versions, so the optimizer accomplishes the same thing, it just takes about 10 times longer.
/Andreas

On 2017-11-24 14:10, Andreas Ekstrand wrote:

Quote:

Hi,

Some more results from my investigations:

Back in February, the MergeGeometryVisitor was changed to be applied on Group instead of Geode by Jannik Heller (in CC). This has resulted in substantially longer optimization time of my loaded OpenFlight files with a large number of polygons. I assume it's a general performance problem with many geodes/geometries.

The removeChild call from MergeGeometryVisitor ::mergeGroup is the main culprit and takes about 87% of the time:
group.removeChild(*ditr);

Does Robert or Jannik have any idea about this? I don't feel comfortable enough with the code to make any changes, but this prevents me from upgrading my software to the latest OSG version, while I can't go back either since I need other fixes in 3.5.8. So I'm a bit stuck and would appreciate any help!

Regards,
Andreas

On 2017-11-19 17:13, Andreas Ekstrand wrote:

Quote:

Hi Robert,

Yes, the model is ineffective in the sense that it has 150 000 separate triangles on the same level in the scene graph, that's the nature of basic usage of OpenFlight and I guess that's why the plugin applies an optimization of its own. But this could be optimized and merged much faster before and I'm just looking for a way to get it to behave like that again, to keep my software from being slower in a new version.

Some results from VTune, where I have manually filtered and presented the top two culprits in 3.5.1 compared to 3.5.8 (where I gave up after 2 minutes):

The MergeGeometryVisitor was applied in 3.5.1 as well but didn't take more than about 0.06s. I guess the 3.5.1 results are generally less interesting but at least they show that MergeGeometryVisitor was not a problem there.

I don't know if this gives you some ideas directly or if we need to keep digging. I will have another look shortly either way, but of course hoping for you or someone else to come up with some bright ideas to shorten the time to fixing this problem.

Regards,
Andreas

On 2017-11-19 16:28, Robert Osfield wrote:

Quote:

Quote:

HI Andreas,

I haven't had a chance to dig further.

One curious thing I noticed is that when I enabled verbose debug
output there was lots of buffer objects being created and destroyed
during the optimisation step. osg::Drawable now assigns
VetextArrayState and VBO's by default for osg::Geometry - this is
required to make VAO support possible. The large number of these
operations suggest lots of creation and merging of osg::Geometry for
this dataset. There is chance tthat this change alone may be causing
a performance slow down.

However, I don't think creation/deletion of osg::Geometry is the crux
of the problem, and may not even be the optimization step - it could
well be down to the nature of the scene graph being created by the
OpenFlight plugin for this dataset. I suspect the source data is
stored in a way that is really inefficient to handle for real-time.
It might be that the OpenFlight plugin just isn't handling the data
well.

As a general rule, running the Optimizer is a last resort for fixing
really bad datasets, ideally datasets should be created in a form that
is appropriate for decent performance right from the start.

Forgot to mention that the geometry before and after optimization is
identical in the different OSG versions, so the optimizer accomplishes the
same thing, it just takes about 10 times longer.

I am currently investigating the issue, currently waiting on valgrind
--tool callgrind to complete to see what clues gives me. It's already
been running quite a while so am getting on with other avenues of
investigation while I wait.

The changes to Optimizer.cpp that you sent to osg-submissions suggests
that it's the removeChild() that is the performance bottleneck. The
changes you've proposed don't handle the case where a group can have
non Drawable children so can't be merged as is. It's still a
possibility of merging it with mods to address this.

I'd like to get down to route cause of the bottleneck though, my best
guess right now is it's an issue of parts of the implementation
triggering a O(N^2) behaviour, i.e. large number of children being
merged. It might be this can be resolved with the approach you took,
or it might be best adding better removeChildren() support that avoids
the O(N^2) behaviour, such as passing a list of nodes to remove.

What this tells us that this dataset generated by the OpenFlight
loader is truly dreadful. It's just appalling how it's handling the
house_150k.flt database. I don't know if this can be traced back to
what the OSG's OpenFlight plugin is doing or whether the tool that
generated the .flt file is just has a completely awful export tool.
What I can say is that the fact that the OSG can handle at all such a
brain dead messed up dataset is a miracle.

The best way to resolve this issue is to refactor the
MergeGeometryVisitor so it handles these really bad datasets with more
grace, avoiding the pitfalls of an O(N^2) algorithm.

Having the core OSG "fix" really bad data is not a proper resolution.
it doesn't fix the fact the the OpenFlight loader is generating such a
dreadful scene graph. It might be that the OpenFlight loader is just
doing what the source database is tell it to do and isn't to blame,
but like the change MergeGeometryVisitor, it may be worth catching
this dreadful datacase on load and never generating the broken scene
graph. It's better to fix problems like this early rather than let it
get all the way to the osgUtil::Optimizer to patch fix things up, the
early it is done the less memory will be used and less memory will be
fragmented and better the performance will be.

I am afraid I just don't have the time to go chasing fixes to awful
3rd party data, I already have enough core OSG issues to do that
unpaid without taking on more. I will resolve the
MergeGeometryVisitor O(N^2) issue but improving the OpenFlight loader
to handle dreadful 3rd party data better I'll have to defer to the
community.

Yes, you can argue that the dataset is awful, but it's actually just storing 150k triangles on one graph level, as separate OpenFlight face records, which is what the OpenFlight file format offers if you don't use its specific mesh nodes that aren't supported by all tools and don't offer full control of each triangle. OpenFlight is a modeling file format which is surely not visualization-friendly. I think the OpenFlight plugin simply uses the Optimizer to convert this to a dataset for effective visualization instead of converting to a more optimal scene graph to start with - a choice that I can't really say if it's good or bad, but the Optimizer should of course handle bad data in the best way possible.

The main issue for me though, is that OSG handles this dataset in a much slower way than before, hence my re-introduction of the previous optimization. But you're completely right, and I realize now, that other nodes than drawables aren't handle correctly. I haven't gotten used to drawables being nodes yet...

So the fact that the OpenFlight plugin creates a non-optimal scene graph and then uses the Optimizer itself to fix this is probably a separate and larger problem - if it indeed is a problem. But I believe it's necessary to at least get MergeGeometryVisitor to behave as fast as before, either by modifying my fixes or by rewriting parts of it. I hope you will find the time to decide on this soon, and please let me know if I can help!

Regards,
Andreas

On 2017-11-27 14:27, Robert Osfield wrote:

Quote:

Quote:

HI Andreas,

I gave up the valgrind tool=callgrind because it was taking too long
to complete. What I did gleen from it was that the run confirms that
removeChildren() is a bottleneck.

To get some context to why it's a bottlenck I put in some stats output
from the MergeGeometryVisitor::mergeGroup(..) and got the following
output:

What this tells us that this dataset generated by the OpenFlight
loader is truly dreadful. It's just appalling how it's handling the
house_150k.flt database. I don't know if this can be traced back to
what the OSG's OpenFlight plugin is doing or whether the tool that
generated the .flt file is just has a completely awful export tool.
What I can say is that the fact that the OSG can handle at all such a
brain dead messed up dataset is a miracle.

The best way to resolve this issue is to refactor the
MergeGeometryVisitor so it handles these really bad datasets with more
grace, avoiding the pitfalls of an O(N^2) algorithm.

Having the core OSG "fix" really bad data is not a proper resolution.
it doesn't fix the fact the the OpenFlight loader is generating such a
dreadful scene graph. It might be that the OpenFlight loader is just
doing what the source database is tell it to do and isn't to blame,
but like the change MergeGeometryVisitor, it may be worth catching
this dreadful datacase on load and never generating the broken scene
graph. It's better to fix problems like this early rather than let it
get all the way to the osgUtil::Optimizer to patch fix things up, the
early it is done the less memory will be used and less memory will be
fragmented and better the performance will be.

I am afraid I just don't have the time to go chasing fixes to awful
3rd party data, I already have enough core OSG issues to do that
unpaid without taking on more. I will resolve the
MergeGeometryVisitor O(N^2) issue but improving the OpenFlight loader
to handle dreadful 3rd party data better I'll have to defer to the
community.

I have checked in a refactor of MergeGeometry visitor that makes it
work properly with groups containing any type of nodes, and avoids the
O(N^2) removChild() usage. In the end I took the line of least
resistance and just used the approach of removing all children and
adding back the ndoes.

On 27 November 2017 at 15:56, Andreas Ekstrand
<> wrote:

Quote:

Yes, you can argue that the dataset is awful, but it's actually just storing
150k triangles on one graph level, as separate OpenFlight face records,
which is what the OpenFlight file format offers if you don't use its
specific mesh nodes that aren't supported by all tools and don't offer full
control of each triangle. OpenFlight is a modeling file format which is
surely not visualization-friendly. I think the OpenFlight plugin simply uses
the Optimizer to convert this to a dataset for effective visualization
instead of converting to a more optimal scene graph to start with - a choice
that I can't really say if it's good or bad, but the Optimizer should of
course handle bad data in the best way possible.

No need to argue whether it's awful or not, there isn't any debate
when it comes to real-time performance : this particular database is
stored in the worst possible way for memory utilization and
performance

Use of the osgUtil::Optimizer is a very crude fix for bad databases.
It's far better to just create a scene graph in an efficient form
right from the start. Optimizing bad databases at best leads to
fragment memory, it really is just papering over cracks and is
unlikely to ever produce optimal databases.

Quote:

The main issue for me though, is that OSG handles this dataset in a much
slower way than before, hence my re-introduction of the previous
optimization. But you're completely right, and I realize now, that other
nodes than drawables aren't handle correctly. I haven't gotten used to
drawables being nodes yet...

So the fact that the OpenFlight plugin creates a non-optimal scene graph and
then uses the Optimizer itself to fix this is probably a separate and larger
problem - if it indeed is a problem.

Yep, big topic. I'm familiar with the problems that the OpenFlight
loader generates, but less so with Creator or other tools that
generate the content. I'd much rather the OpenFlight loader builds
more efficient database to start with rather than rely upon the
Optimizer to fix problems once the data is loaded.

Quote:

But I believe it's necessary to at
least get MergeGeometryVisitor to behave as fast as before, either by
modifying my fixes or by rewriting parts of it. I hope you will find the
time to decide on this soon, and please let me know if I can help!

Thanks Robert! Works fine here as well, and I think the line of least resistance was the right way this time.

Any plans on a 3.5.9 release soon? I'm now facing the problem of either waiting until the next release, or basing my software on 3.5.8 with this modification and thus implicitly recommending users to use 3.5.8 to be compatible but with the implication that their OpenFlight models might be load slowly...

/Andreas

On 2017-11-27 17:25, Robert Osfield wrote:

Quote:

Quote:

Hi Andreas,

I have checked in a refactor of MergeGeometry visitor that makes it
work properly with groups containing any type of nodes, and avoids the
O(N^2) removChild() usage. In the end I took the line of least
resistance and just used the approach of removing all children and
adding back the ndoes.

On 27 November 2017 at 15:56, Andreas Ekstrand
<> (

Only registered users can see emails on this board!Get registred or enter the forums!

) wrote:

Quote:

Yes, you can argue that the dataset is awful, but it's actually just storing
150k triangles on one graph level, as separate OpenFlight face records,
which is what the OpenFlight file format offers if you don't use its
specific mesh nodes that aren't supported by all tools and don't offer full
control of each triangle. OpenFlight is a modeling file format which is
surely not visualization-friendly. I think the OpenFlight plugin simply uses
the Optimizer to convert this to a dataset for effective visualization
instead of converting to a more optimal scene graph to start with - a choice
that I can't really say if it's good or bad, but the Optimizer should of
course handle bad data in the best way possible.

No need to argue whether it's awful or not, there isn't any debate
when it comes to real-time performance : this particular database is
stored in the worst possible way for memory utilization and
performance

Use of the osgUtil::Optimizer is a very crude fix for bad databases.
It's far better to just create a scene graph in an efficient form
right from the start. Optimizing bad databases at best leads to
fragment memory, it really is just papering over cracks and is
unlikely to ever produce optimal databases.

Quote:

The main issue for me though, is that OSG handles this dataset in a much
slower way than before, hence my re-introduction of the previous
optimization. But you're completely right, and I realize now, that other
nodes than drawables aren't handle correctly. I haven't gotten used to
drawables being nodes yet...

So the fact that the OpenFlight plugin creates a non-optimal scene graph and
then uses the Optimizer itself to fix this is probably a separate and larger
problem - if it indeed is a problem.

Yep, big topic. I'm familiar with the problems that the OpenFlight
loader generates, but less so with Creator or other tools that
generate the content. I'd much rather the OpenFlight loader builds
more efficient database to start with rather than rely upon the
Optimizer to fix problems once the data is loaded.

Quote:

But I believe it's necessary to at
least get MergeGeometryVisitor to behave as fast as before, either by
modifying my fixes or by rewriting parts of it. I hope you will find the
time to decide on this soon, and please let me know if I can help!

Only registered users can see emails on this board!Get registred or enter the forums!

) wrote:

Quote:

Thanks Robert! Works fine here as well, and I think the line of least
resistance was the right way this time.

Any plans on a 3.5.9 release soon? I'm now facing the problem of either
waiting until the next release, or basing my software on 3.5.8 with this
modification and thus implicitly recommending users to use 3.5.8 to be
compatible but with the implication that their OpenFlight models might be
load slowly...

I was planning to tag 3.5.9 in the next day or so. There isn't any
major feature I'm waiting on, so as long as it compiles and has no
regressions I can tag it right away,

You cannot post new topics in this forumYou cannot reply to topics in this forumYou cannot edit your posts in this forumYou cannot delete your posts in this forumYou cannot vote in polls in this forumYou cannot attach files in this forumYou cannot download files in this forum