<div dir="ltr">Would it make sense to custom hack a BUNCH of asserts into the suspect code to validate all possible assumptions at runtime and maybe pinpoint erroneous conditions prior to hitting the actual crash?</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jan 22, 2019 at 9:42 PM Robert Osfield <<a href="mailto:robert.osfield@gmail.com">robert.osfield@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Richard,<br>
<br>
Sorry to hear your are battling this issue. I've read through, and<br>
had a quick look at simgear master but could find the<br>
loadUsingReaderWriter() implementation in ModelRegistry.cxx that you<br>
mention. Are you working on a branch or not checked something in yet?<br>
<br>
As a general comment, if you could use the OpenSceneGraph-3.6 branch<br>
rather than master it should give you a more stable and tested OSG<br>
base to work from. I'd also recommend checking the whole model<br>
loading/processing path to make sure all the methods are taking and<br>
passing back ref_ptr<> rather than C pointer. In my quick check I<br>
spotted a osg::Node* OptimizeModelPolicy::optimize() method that be<br>
something could be the type of thing to convert across to ref_ptr<>.<br>
<br>
Robert.<br>
<br>
On Tue, 22 Jan 2019 at 17:58, Richard Harrison <<a href="mailto:rjh@zaretto.com" target="_blank">rjh@zaretto.com</a>> wrote:<br>
><br>
> I've just got another of these problems.<br>
><br>
> This is after changing all of the osgDB::read into osgDB::readRef<br>
> (simgear commit cb024dd82d4c384df0b599640a98e762fbf66688) and 5days of<br>
> flight time testing (not all the same run, FG was restarted many<br>
> times) I've hit what looks like a the same problem as I originally<br>
> reported; i.e. the expiry appears to be something that has just been<br>
> loaded and expired at the same time. I'm keeping my debug session open<br>
> to allow further investigation in case questions.<br>
><br>
> I'm surprised that the fixes didn't work as they looked to me as<br>
> though they should fix the problem I'm immediately suspecting that<br>
> maybe there are other things that we're doing that are interfering<br>
> with the thread safety mechanisms.<br>
><br>
> Having dug into what's happening; the DatabasePager is currently<br>
> loading Models/Airport/cargoim.xml; which is defined in<br>
> project3000/Objects/w010n50/w002n52/2925458.stg; and the ObjectCache<br>
> is expiring Models/Aircraft/Cessna172_red.ac; Looking at the pertinent<br>
> part of the .stg it is a fair conclusion that the DatabasePager has<br>
> just loaded two Cessna172_red.ac models<br>
><br>
> OBJECT_SHARED Models/Aircraft/Cessna172_red.ac -1.47630 52.37373 82.02<br>
> 223.53<br>
> OBJECT_SHARED Models/Aircraft/Cessna172_red.ac -1.47560 52.37443 81.34 345.5<br>
> OBJECT_SHARED Models/lib/<a href="http://trailer-fedex.ac" rel="noreferrer" target="_blank">trailer-fedex.ac</a> -1.48893 52.36957 84.13 314.01<br>
> OBJECT_SHARED Models/Airport/cargoim.xml -1.47436 52.36886 79.91 188.47<br>
><br>
> The actual .ac model load is happening in SGReaderWriteXML.cxx line 341<br>
><br>
> modelResult = osgDB::readRefNodeFile(modelpath.local8BitStr(),<br>
> options.get());<br>
><br>
> which will end up in ModelRegistry.cxx line 866<br>
><br>
> loadUsingReaderWriter(const std::string& fileName,<br>
> const osgDB::Options* opt)<br>
> {<br>
> using namespace osgDB;<br>
> ReaderWriter* rw =<br>
> Registry::instance()<br>
> ->getReaderWriterForExtension<br>
> (osgDB::getFileExtension(fileName));<br>
> if (!rw)<br>
> return ReaderWriter::ReadResult(); // FILE_NOT_HANDLED<br>
> return rw->readNode(fileName, opt);<br>
> }<br>
><br>
> I think it is correct in this instance to use the (ac3d) via the<br>
> registry and readNode.<br>
><br>
> The only other thing that looks a bit odd is the way we are requesting<br>
> the same .stg file multiple times; maybe that is tripping something up<br>
> in our code; but I don't think that's the cause of the deleting whilst<br>
> still in use.<br>
><br>
> [0..162] i:/flightgear/project3000/Objects/w010n50/w002n52/2925458.stg<br>
> [163..178] i:/flightgear/project3000/Objects/w010n50/w002n52/2925458.stg<br>
> [179..] i:/flightgear/terrasync/Terrain/w010n50/w002n52/2925458.stg<br>
><br>
><br>
> -------------------<br>
> On 17/01/2019 14:39, Voerman, L. wrote the following questions:<br>
><br>
> > - did the problematic node come out of the cache, or did it come<br>
> fresh from disk?<br>
><br>
> It's hard to tell because as far as I can tell the problematic load has<br>
> finished and the pager has moved onto the next item.<br>
><br>
> > - Is the parent group (and it's _children vector) still sane?<br>
><br>
> Looking at the node that is being expired it all looks good; the<br>
> reference count is 3; so there remains the mystery of how this can<br>
> happen.<br>
><br>
> oitr (<br>
> ("Models/Aircraft/Cessna172_<a href="http://red.ac" target="_blank">red.ac</a>",<br>
> {_ptr=0x0 <NULL> }),<br>
> ({_ptr=0x2441e8bc800 {_children={ size=0x5 } } },<br>
> 5002.8604968999998))<br>
> first "Models/Aircraft/Cessna172_<a href="http://red.ac" target="_blank">red.ac</a>"<br>
> second {_ptr=0x0 <NULL> }<br>
> second ({_ptr=0x2441e8bc800 {_children={ size=0x5 } } },<br>
> 5002.8604968999998)<br>
> first {_ptr=0x2441e8bc800 {_children={ size=0x5 } } } o<br>
> sg::ref_ptr<osg::Object><br>
> _ptr 0x2441e8bc800 {_children={ size=0x5 } }<br>
> osg::Object * {osg160-osg.dll!osg::Group}<br>
> [osg::Group]<br>
> {_children={ size=0x5 } } osg160-osg.dll!osg::Group<br>
> osg::Node {...}<br>
> osg::Object<br>
> {<br>
> _name="I:\flightgear\terrasync\Models\Aircraft\Cessna172_red.ac"<br>
> _dataVariance=STATIC (0x1) ...<br>
> }<br>
> _initialBound<br>
> {_center={_v=0x2441e8bc848 {0.0, 0.0, 0.0} } _radius=-1.0 }<br>
> _boundingSphere {<br>
> _center={_v=0x2441e8bc860 {0.0, 0.0, 0.0} }<br>
> _radius=-1.0 }<br>
> _boundingSphereComputed false bool<br>
> _parents { size=0x2 }<br>
> _numChildrenRequiringUpdateTraversal<br>
> _cullingActive true<br>
> _nodeMask 0xffffdfff<br>
> _children { size=0x5 }<br>
> osg::Referenced {_observerSet={_ptr=0x0 }<br>
> _refCount={_value=0x3 } }<br>
> _dataVariance STATIC (0x1)<br>
> _userDataContainer 0x0<br>
> second 5002.8604968999998<br>
><br>
> > - If the parent node is still sane, can you match it to the file on<br>
> > disk and possibly tell what sort of node the problem appears in? -<br>
> > What is the file format of the file on disk? Do you have (use)<br>
> > multiple pager threads? Could the file loader have a multithreading<br>
> > problem?<br>
><br>
> It's a .ac format node; we're using a single DatabasePager thread (as<br>
> there are other loading problems within FG that almost prevent it from<br>
> running at all)<br>
><br>
> --------------------------<br>
> Stack dumps are as follows:<br>
><br>
> Main thread;<br>
> fgfs.exe!NotifyLogger::notify<br>
> osg160-osg.dll!osg::NotifyStreamBuffer::sync L:92<br>
> msvcp140.dll!00007ffabb8a27f2<br>
> osg160-osg.dll!std::endl<br>
> osg160-osg.dll!osg::Referenced::~Referenced() L:205<br>
> osgdb_ac.dll!osg::Group::`scalar deleting destructor'<br>
> osg160-osg.dll!osg::Referenced::signalObserversAndDelete() L:294<br>
> osg160-osg.dll!osg::Referenced::unref() L:203<br>
> osg160-osgDB.dll!std::erase L:1431<br>
> osg160-osgDB.dll!osgDB::ObjectCache::removeExpiredObjectsInCache L:171<br>
> osg160-osgViewer.dll!osgViewer::Viewer::updateTraversal() L:1161<br>
> osg160-osgViewer.dll!osgViewer::ViewerBase::frame L:748<br>
> fgfs.exe!fgOSMainLoop() L:339<br>
> fgfs.exe!fgMainInit(int argc, char * * argv) L:619<br>
> fgfs.exe!main(int argc, char * * argv) L:339<br>
> fgfs.exe!__scrt_common_main_seh() L:253<br>
><br>
> DatabasePager thread<br>
><br>
> ntdll.dll!00007ffaefd51db4<br>
> KernelBase.dll!00007ffaec855834<br>
> KernelBase.dll!00007ffaec855bcc<br>
> kernel32.dll!00007ffaef606413<br>
> kernel32.dll!00007ffaef642d42<br>
> osg160-osgDB.dll!osgDB::getRealPath<br>
> osg160-osgDB.dll!osgDB::findFileInPath<br>
> osg160-osgDB.dll!osgDB::findFileInPath<br>
> osg160-osgDB.dll!osgDB::Registry::findDataFileImplementation<br>
> osg160-osgDB.dll!osgDB::Registry::findDataFile<br>
> osg160-osgDB.dll!osgDB::findDataFile<br>
> fgfs.exe!simgear::SGReaderWriterXML::readNode<br>
> fgfs.exe!simgear::ModelRegistryCallback<>::loadUsingReaderWriter<br>
> fgfs.exe!simgear::ModelRegistryCallback<>::readNode<br>
> fgfs.exe!simgear::ModelRegistry::readNode<br>
> osg160-osgDB.dll!osgDB::Registry::readNode<br>
> osg160-osgDB.dll!osgDB::readRefNodeFile<br>
> fgfs.exe!simgear::ReaderWriterSTG::_ModelBin::DelayLoadReadFileCallback<br>
> ::AddModelLOD::operator<br>
> fgfs.exe!simgear::QuadTreeBuilder<>::addNode<br>
> fgfs.exe!simgear::QuadTreeBuilder<>::buildQuadTree<br>
> fgfs.exe!simgear::ReaderWriterSTG::_ModelBin<br>
> ::DelayLoadReadFileCallback::readNode<br>
> osg160-osgDB.dll!osgDB::Registry::readNode<br>
> osg160-osgDB.dll!osgDB::DatabasePager::DatabaseThread::run<br>
> ot21-OpenThreads.dll!OpenThreads::ThreadPrivateActions::StartThread<br>
><br>
> _______________________________________________<br>
> osg-users mailing list<br>
> <a href="mailto:osg-users@lists.openscenegraph.org" target="_blank">osg-users@lists.openscenegraph.org</a><br>
> <a href="http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org" rel="noreferrer" target="_blank">http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org</a><br>
_______________________________________________<br>
osg-users mailing list<br>
<a href="mailto:osg-users@lists.openscenegraph.org" target="_blank">osg-users@lists.openscenegraph.org</a><br>
<a href="http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org" rel="noreferrer" target="_blank">http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div style="text-align:center">Chris 'Xenon' Hanson, omo sanza lettere. Xenon@AlphaPixel.com <a href="http://www.alphapixel.com/" target="_blank">http://www.alphapixel.com/</a></div><div style="text-align:center">Training • Consulting • Contracting</div><div style="text-align:center">3D • Scene Graphs (Open Scene Graph/OSG) • OpenGL 2 • OpenGL 3 • OpenGL 4 • GLSL • OpenGL ES 1 • OpenGL ES 2 • OpenCL</div><div style="text-align:center"><span style="font-size:12.8px">Legal/IP •</span><span style="font-size:12.8px"> </span><span style="font-size:12.8px">Forensics •</span><span style="font-size:12.8px"> </span>Imaging <span style="font-size:12.8px">•</span><span style="font-size:12.8px"> </span><span style="font-size:12.8px">UAVs </span><span style="font-size:12.8px">• GIS • GPS • osgEarth • Terrain • Telemetry • Cryptography • LIDAR • Embedded • Mobile • iPhone/iPad/iOS • Android</span></div><div style="text-align:center"><a href="https://twitter.com/alphapixel" target="_blank">@alphapixel</a> <a href="http://facebook.com/alphapixel" target="_blank">facebook.com/alphapixel</a> (775) 623-PIXL [7495]<br></div></div></div></div></div></div>