[osg-users] Stencil and render to texture

[Nicolas Baillard]

Hello everyone.

I'm having trouble using the stencil buffer while doing a render to texture. My purpose is to apply a black mask in front of my scene, as if I was looking at the world through a crack in a wall. I managed to achieve this effect using the stencil before. But back then I was doing direct rendering on the screen. Now I'm trying to render to a texture and it doesn't work any more.

Here is what I do:
First I create the mask I want to draw into the stencil buffer. Here it's a simple quad. I set a stencil function on it so it writes 1 into the stencil buffer for each pixel.

Code:

osg::Vec3Array *vertices = new osg::Vec3Array();
vertices->push_back(osg::Vec3(0.25, 0.25, 0.0));
vertices->push_back(osg::Vec3(0.75, 0.25, 0.0));
vertices->push_back(osg::Vec3(0.75, 0.75, 0.0));
vertices->push_back(osg::Vec3(0.25, 0.75, 0.0));
_stencil_geometry = new osg::Geometry();
_stencil_geometry->setVertexArray(vertices);
_stencil_geometry->addPrimitiveSet(new osg::DrawArrays(GL_QUADS, 0, 4));
_stencil_node = new osg::Geode();
_stencil_node->addDrawable(_stencil_geometry);
_stencil_node->getOrCreateStateSet()->setMode(GL_LIGHTING, osg::StateAttribute::OFF);
_stencil_node->getOrCreateStateSet()->setMode(GL_DEPTH_TEST, osg::StateAttribute::OFF);
osg::Stencil* stencilSet = new osg::Stencil();
stencilSet->setFunction(osg::Stencil::ALWAYS, 1, ~0u);
stencilSet->setOperation(osg::Stencil::REPLACE, osg::Stencil::REPLACE, osg::Stencil::REPLACE);
_stencil_node->getOrCreateStateSet()->setAttributeAndModes(stencilSet, osg::StateAttribute::ON | osg::StateAttribute::OVERRIDE);




Then I create a camera to render my mask into the stencil buffer. I make it so that this camera clears the whole stencil to 0 before rendering.

Code:

_stencil_camera = new osg::Camera();
_stencil_camera->setViewport(0, 0, _w, _h);
_stencil_camera->setRenderOrder(osg::Camera::PRE_RENDER, 1);
_stencil_camera->setClearMask(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
_stencil_camera->setClearColor(Color(0., 0., 0., 1.));
_stencil_camera->setClearStencil(0);
_stencil_camera->setReferenceFrame(osg::Transform::ABSOLUTE_RF);
_stencil_camera->setProjectionMatrixAsOrtho2D(0., 1., 0., 1.);
_stencil_camera->setRenderTargetImplementation(osg::Camera::FRAME_BUFFER_OBJECT);
_stencil_camera->attach(osg::Camera::PACKED_DEPTH_STENCIL_BUFFER, GL_DEPTH_STENCIL_EXT);
_stencil_camera->addChild(_stencil_node);




Finally I create the camera that will render my scene and I attach the stencil camera to it as a child. I set the render order so that this camera is rendered after the stencil camera. I configure this camera to render into a texture and I add it to the view.

Code:

_display_buffer = new osg::Texture2D();
_display_buffer->setTextureSize(_w, _h);
_display_buffer->setInternalFormat(GL_RGB);
_display_buffer->setFilter(osg::Texture2D::MIN_FILTER, _min_filter_mode);
_display_buffer->setFilter(osg::Texture2D::MAG_FILTER, _mag_filter_mode);

_display_camera = new osg::Camera();
_display_camera->setGraphicsContext(context);
_display_camera->setViewport(0, 0, _w, _h);
_display_camera->setRenderOrder(osg::Camera::PRE_RENDER, 2);
_display_camera->setClearColor(Color(0., 0., 0., 1.));
_display_camera->setClearMask(0);
_display_camera->setReferenceFrame(osg::Transform::ABSOLUTE_RF);
_display_camera->addChild(_stencil_camera);
_display_camera->addChild(_my_scene_root_node);
_display_camera->setProjectionMatrixAsOrtho2D(0., 1., 0., 1.);
_display_camera->setRenderTargetImplementation(osg::Camera::FRAME_BUFFER_OBJECT);
_display_camera->attach(osg::Camera::COLOR_BUFFER, _display_buffer);
_display_camera->attach(osg::Camera::PACKED_DEPTH_STENCIL_BUFFER, GL_DEPTH_STENCIL_EXT);
_view->addSlave(_display_camera, false);




Finally I set a stencil test on the whole scene. The scene shall only be drawn on pixels where the stencil is not 0 (so whereever my mask was drawn).

Code:

osg::Stencil* stencilTest = new osg::Stencil();
stencilTest->setFunction(osg::Stencil::NOTEQUAL, 0, ~0u);
stencilTest->setOperation(osg::Stencil::KEEP, osg::Stencil::KEEP, osg::Stencil::KEEP);
_my_scene_root_node->getOrCreateStateSet()->setAttributeAndModes(stencilTest, osg::StateAttribute::ON);




Well, this doesn't work. All I get is a black texture (or whatever clear color I set on my display camera). When I do the very same thing without the render to texture part it works. But with the render to texture, it's as if my mask was never drawn into the stencil buffer. What's weird is that if I configure my stencil camera to clear the stencil buffer to a value different that 0 then my scene is rendered to the texture. So it proves that the stencil works. But for some reasons I don't understand my mask seems to never be drawn into it.

Could someone please tell me what I did wrong ?

Regards,
Nicolas

------------------
Read this topic online here:
http://forum.openscenegraph.org/viewtopic.php?p=65335#65335