[osg-users] Slow optimization and OpenFlight

[Robert Osfield]

HI Andreas,

I gave up the valgrind tool=callgrind because it was taking too long
to complete.  What I did gleen from it was that the run confirms that
removeChildren() is a bottleneck.

To get some context to why it's a bottlenck I put in some stats output
from the MergeGeometryVisitor::mergeGroup(..) and got the following
output:

$ osgconv house_150k.flt test3.osgb
Optimizer::MergeGeometryVisitor::mergeGroup() 148976
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=3333
    duplicateList.size()=2324
Data written to 'test3.osgb'.

What this tells us that this dataset generated by the OpenFlight
loader is truly dreadful.  It's just appalling how it's handling the
house_150k.flt database.  I don't know if this can be traced back to
what the OSG's OpenFlight plugin is doing or whether the tool that
generated the .flt file is just has a completely awful export tool.
What I can say is that the fact that the OSG can handle at all such a
brain dead messed up dataset is a miracle.

The best way to resolve this issue is to refactor the
MergeGeometryVisitor so it handles these really bad datasets with more
grace, avoiding the pitfalls of an O(N^2) algorithm.

Having the core OSG "fix" really bad data is not a proper resolution.
it doesn't fix the fact the the OpenFlight loader is generating such a
dreadful scene graph.  It might be that the OpenFlight loader is just
doing what the source database is tell it to do and isn't to blame,
but like the change MergeGeometryVisitor, it may be worth catching
this dreadful datacase on load and never generating the broken scene
graph.  It's better to fix problems like this early rather than let it
get all the way to the osgUtil::Optimizer to patch fix things up, the
early it is done the less memory will be used and less memory will be
fragmented and better the performance will be.

I am afraid I just don't have the time to go chasing fixes to awful
3rd party data, I already have enough core OSG issues to do that
unpaid without taking on more.  I will resolve the
MergeGeometryVisitor O(N^2) issue but improving the OpenFlight loader
to handle dreadful 3rd party data better I'll have to defer to the
community.

Cheers,
Robert.