If you're running AdBlock, please consider whitelisting this site if you'd like to support LearnOpenGL (it helps a lot); and no worries, I won't be mad if you don't :)

1. Introduction
2. Getting started
   1. OpenGL
   2. Creating a window
   3. Hello Window
   4. Hello Triangle
   5. Shaders
   6. Textures
   7. Transformations
   8. Coordinate Systems
   9. Camera
   10. Review
3. Lighting
   1. Colors
   2. Basic Lighting
   3. Materials
   4. Lighting maps
   5. Light casters
   6. Multiple lights
   7. Review
4. Model Loading
   1. Assimp
   2. Mesh
   3. Model
5. Advanced OpenGL
   1. Depth testing
   2. Stencil testing
   3. Blending
   4. Face culling
   5. Framebuffers
   6. Cubemaps
   7. Advanced Data
   8. Advanced GLSL
   9. Geometry Shader
   10. Instancing
   11. Anti Aliasing
6. Advanced Lighting
   1. Advanced Lighting
   2. Gamma Correction
   3. Shadows
      1. Shadow Mapping
      2. Point Shadows
   4. Normal Mapping
   5. Parallax Mapping
   6. HDR
   7. Bloom
   8. Deferred Shading
   9. SSAO
7. PBR
   1. Theory
   2. Lighting
   3. IBL
      1. Diffuse irradiance
      2. Specular IBL
8. In Practice
   1. Debugging
   2. Text Rendering
   3. 2D Game
      1. Breakout
      2. Setting up
      3. Rendering Sprites
      4. Levels
      5. Collisions
         1. Ball
         2. Collision detection
         3. Collision resolution
      6. Particles
      7. Postprocessing
      8. Powerups
      9. Audio
      10. Render text
      11. Final thoughts
9. Guest Articles
   1. How to publish
   2. 2020
      1. OIT
         1. Introduction
         2. Weighted Blended
      2. Skeletal Animation
   3. 2021
      1. CSM
      2. Scene
         1. Scene Graph
         2. Frustum Culling
      3. Tessellation
         1. Height map
         2. Tessellation
      4. DSA
   4. 2022
      1. Compute Shaders
         1. Introduction
      2. Phys. Based Bloom
      3. Area Lights
10. Code repository
11. Translations
12. Privacy
13. About

Stencil testing

Advanced-OpenGL/Stencil-testing

Once the fragment shader has processed the fragment a so called stencil test is executed that, just like the depth test, has the option to discard fragments. After that the remaining fragments are passed to the depth test where OpenGL could possibly discard even more fragments. The stencil test is based on the content of yet another buffer called the stencil buffer that we're allowed to update during rendering to achieve interesting effects.

A stencil buffer (usually) contains 8 bits per stencil value that amounts to a total of 256 different stencil values per pixel. We can set these stencil values to values of our liking and we can discard or keep fragments whenever a particular fragment has a certain stencil value.

Each windowing library needs to set up a stencil buffer for you. GLFW does this automatically so we don't have to tell GLFW to create one, but other windowing libraries may not create a stencil buffer by default so be sure to check your library's documentation.

A simple example of a stencil buffer is shown below (pixels not-to-scale):

The stencil buffer is first cleared with zeros and then an open rectangle of 1s is stored in the stencil buffer. The fragments of the scene are then only rendered (the others are discarded) wherever the stencil value of that fragment contains a 1.

Stencil buffer operations allow us to set the stencil buffer at specific values wherever we're rendering fragments. By changing the content of the stencil buffer while we're rendering, we're writing to the stencil buffer. In the same (or following) frame(s) we can read these values to discard or pass certain fragments. When using stencil buffers you can get as crazy as you like, but the general outline is usually as follows:

• Enable writing to the stencil buffer.
• Render objects, updating the content of the stencil buffer.
• Disable writing to the stencil buffer.
• Render (other) objects, this time discarding certain fragments based on the content of the stencil buffer.

By using the stencil buffer we can thus discard certain fragments based on the fragments of other drawn objects in the scene.

You can enable stencil testing by enabling GL_STENCIL_TEST. From that point on, all rendering calls will influence the stencil buffer in one way or another.

glEnable(GL_STENCIL_TEST);    

Note that you also need to clear the stencil buffer each iteration just like the color and depth buffer:

glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT); 

Also, just like the depth testing's glDepthMask function, there is an equivalent function for the stencil buffer. The function glStencilMask allows us to set a bitmask that is ANDed with the stencil value about to be written to the buffer. By default this is set to a bitmask of all 1s not affecting the output, but if we were to set this to 0x00 all the stencil values written to the buffer end up as 0s. This is equivalent to depth testing's glDepthMask(GL_FALSE):

glStencilMask(0xFF); // each bit is written to the stencil buffer as is
glStencilMask(0x00); // each bit ends up as 0 in the stencil buffer (disabling writes)

Most of the cases you'll only be using 0x00 or 0xFF as the stencil mask, but it's good to know there are options to set custom bit-masks.

Stencil functions

Similar to depth testing, we have a certain amount of control over when a stencil test should pass or fail and how it should affect the stencil buffer. There are a total of two functions we can use to configure stencil testing: glStencilFunc and glStencilOp.

The glStencilFunc(GLenum func, GLint ref, GLuint mask) has three parameters:

• func: sets the stencil test function that determines whether a fragment passes or is discarded. This test function is applied to the stored stencil value and the glStencilFunc's ref value. Possible options are: GL_NEVER, GL_LESS, GL_LEQUAL, GL_GREATER, GL_GEQUAL, GL_EQUAL, GL_NOTEQUAL and GL_ALWAYS. The semantic meaning of these is similar to the depth buffer's functions.
• ref: specifies the reference value for the stencil test. The stencil buffer's content is compared to this value.
• mask: specifies a mask that is ANDed with both the reference value and the stored stencil value before the test compares them. Initially set to all 1s.

So in the case of the simple stencil example we've shown at the start, the function would be set to:

glStencilFunc(GL_EQUAL, 1, 0xFF)

This tells OpenGL that whenever the stencil value of a fragment is equal (GL_EQUAL) to the reference value 1, the fragment passes the test and is drawn, otherwise discarded.

But glStencilFunc only describes whether OpenGL should pass or discard fragments based on the stencil buffer's content, not how we can actually update the buffer. That is where glStencilOp comes in.

The glStencilOp(GLenum sfail, GLenum dpfail, GLenum dppass) contains three options of which we can specify for each option what action to take:

• sfail: action to take if the stencil test fails.
• dpfail: action to take if the stencil test passes, but the depth test fails.
• dppass: action to take if both the stencil and the depth test pass.

Then for each of the options you can take any of the following actions:

Action	Description
GL_KEEP	The currently stored stencil value is kept.
GL_ZERO	The stencil value is set to 0.
GL_REPLACE	The stencil value is replaced with the reference value set with glStencilFunc.
GL_INCR	The stencil value is increased by 1 if it is lower than the maximum value.
GL_INCR_WRAP	Same as GL_INCR, but wraps it back to 0 as soon as the maximum value is exceeded.
GL_DECR	The stencil value is decreased by 1 if it is higher than the minimum value.
GL_DECR_WRAP	Same as GL_DECR, but wraps it to the maximum value if it ends up lower than 0.
GL_INVERT	Bitwise inverts the current stencil buffer value.

By default the glStencilOp function is set to (GL_KEEP, GL_KEEP, GL_KEEP) so whatever the outcome of any of the tests, the stencil buffer keeps its values. The default behavior does not update the stencil buffer, so if you want to write to the stencil buffer you need to specify at least one different action for any of the options.

So using glStencilFunc and glStencilOp we can precisely specify when and how we want to update the stencil buffer and when to pass or discard fragments based on its content.

Object outlining

It would be unlikely if you completely understood how stencil testing works from the previous sections alone so we're going to demonstrate a particular useful feature that can be implemented with stencil testing alone called object outlining.

Object outlining does exactly what it says it does. For each object (or only one) we're creating a small colored border around the (combined) objects. This is a particular useful effect when you want to select units in a strategy game for example and need to show the user which of the units were selected. The routine for outlining your objects is as follows:

1. Enable stencil writing.
2. Set the stencil op to GL_ALWAYS before drawing the (to be outlined) objects, updating the stencil buffer with 1s wherever the objects' fragments are rendered.
3. Render the objects.
4. Disable stencil writing and depth testing.
5. Scale each of the objects by a small amount.
6. Use a different fragment shader that outputs a single (border) color.
7. Draw the objects again, but only if their fragments' stencil values are not equal to 1.
8. Enable depth testing again and restore stencil func to GL_KEEP.

This process sets the content of the stencil buffer to 1s for each of the object's fragments and when it's time to draw the borders, we draw scaled-up versions of the objects only where the stencil test passes. We're effectively discarding all the fragments of the scaled-up versions that are part of the original objects' fragments using the stencil buffer.

So we're first going to create a very basic fragment shader that outputs a border color. We simply set a hardcoded color value and call the shader shaderSingleColor:

void main()
{
    FragColor = vec4(0.04, 0.28, 0.26, 1.0);
}

Using the scene from the previous chapter we're going to add object outlining to the two containers, so we'll leave the floor out of it. We want to first draw the floor, then the two containers (while writing to the stencil buffer), and then draw the scaled-up containers (while discarding the fragments that write over the previously drawn container fragments).

We first need to enable stencil testing:

glEnable(GL_STENCIL_TEST);

And then in each frame we want to specify the action to take whenever any of the stencil tests succeed or fail:

glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);  

If any of the tests fail we do nothing; we simply keep the currently stored value that is in the stencil buffer. If both the stencil test and the depth test succeed however, we want to replace the stored stencil value with the reference value set via glStencilFunc which we later set to 1.

We clear the stencil buffer to 0s at the start of the frame and for the containers we update the stencil buffer to 1 for each fragment drawn:

glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);  
glStencilFunc(GL_ALWAYS, 1, 0xFF); // all fragments should pass the stencil test
glStencilMask(0xFF); // enable writing to the stencil buffer
normalShader.use();
DrawTwoContainers();

By using GL_REPLACE as the stencil op function we make sure that each of the containers' fragments update the stencil buffer with a stencil value of 1. Because the fragments always pass the stencil test, the stencil buffer is updated with the reference value wherever we've drawn them.

Now that the stencil buffer is updated with 1s where the containers were drawn we're going to draw the upscaled containers, but this time with the appropriate test function and disabling writes to the stencil buffer:

glStencilFunc(GL_NOTEQUAL, 1, 0xFF);
glStencilMask(0x00); // disable writing to the stencil buffer
glDisable(GL_DEPTH_TEST);
shaderSingleColor.use(); 
DrawTwoScaledUpContainers();

We set the stencil function to GL_NOTEQUAL to make sure that we're only drawing parts of the containers that are not equal to 1. This way we only draw the part of the containers that are outside the previously drawn containers. Note that we also disable depth testing so the scaled up containers (e.g. the borders) do not get overwritten by the floor. Make sure to enable the depth buffer again once you're done.

The total object outlining routine for our scene looks something like this:

glEnable(GL_DEPTH_TEST);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);  
  
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT); 

glStencilMask(0x00); // make sure we don't update the stencil buffer while drawing the floor
normalShader.use();
DrawFloor()  
  
glStencilFunc(GL_ALWAYS, 1, 0xFF); 
glStencilMask(0xFF); 
DrawTwoContainers();
  
glStencilFunc(GL_NOTEQUAL, 1, 0xFF);
glStencilMask(0x00); 
glDisable(GL_DEPTH_TEST);
shaderSingleColor.use(); 
DrawTwoScaledUpContainers();
glStencilMask(0xFF);
glStencilFunc(GL_ALWAYS, 1, 0xFF);   
glEnable(GL_DEPTH_TEST);  

As long as you understand the general idea behind stencil testing this shouldn't be too hard to understand. Otherwise try to carefully read the previous sections again and try to completely understand what each of the functions does now that you've seen an example of it can be used.

The result of the outlining algorithm then looks like this:

Check the source code here to see the complete code of the object outlining algorithm.

You can see that the borders overlap between both containers which is usually the effect that we want (think of strategy games where we want to select 10 units; merging borders is generally preferred). If you want a complete border per object you'd have to clear the stencil buffer per object and get a little creative with the depth buffer.

The object outlining algorithm you've seen is commonly used in games to visualize selected objects (think of strategy games) and an algorithm like this can easily be implemented within a model class. You could set a boolean flag within the model class to draw either with borders or without. If you want to be creative you could even give the borders a more natural look with the help of post-processing filters like Gaussian Blur.

Stencil testing has many more purposes (beside outlining objects) like drawing textures inside a rear-view mirror so it neatly fits into the mirror shape, or rendering real-time shadows with a stencil buffer technique called shadow volumes. Stencil buffers give us with yet another nice tool in our already extensive OpenGL toolkit.

HI