Anti Aliasing


Somewhere in your adventurous rendering journey you probably came across some jagged saw-like patterns along the edges of your models. The reason these jagged edges appear is due to how the rasterizer transforms the vertex data into actual fragments behind the scene. An example of what these jagged edges look like can already be seen when drawing a simple cube:

Container with visible aliasing

While not immediately visible, if you take a closer look at the edges of the cube you'll see a jagged pattern. If we zoom in you'd see the following:

Zoomed in on contanier with visible aliasing

This is clearly not something we want in a final version of an application. This effect, of clearly seeing the pixel formations an edge is composed of, is called aliasing. There are quite a few techniques out there called anti-aliasing techniques that fight exactly this aliasing behavior to produce more smooth edges.

At first we had a technique called super sample anti-aliasing (SSAA) that temporarily used a much higher resolution to render the scene in (super sampling) and when the visual output is updated in the framebuffer, the resolution was downsampled back to the normal resolution. This extra resolution was used to prevent these jagged edges. While it did provide us with a solution to the aliasing problem, it came with a major performance drawback since we had to draw a lot more fragments than usual. This technique therefore only had a short glory moment.

This technique did gave birth to a more modern technique called multisample anti-aliasing or MSAA that borrows from the concepts behind SSAA while implementing a much more efficient approach. In this tutorial we'll be extensively discussing this MSAA technique that is built-in in OpenGL.


To understand what multisampling is and how it works into solving the aliasing problem we first need to delve a bit further into the inner workings of OpenGL's rasterizer.

The rasterizer is the combination of all algorithms and processes that sit between your final processed vertices and the fragment shader. The rasterizer takes all vertices belonging to a single primitive and transforms this to a set of fragments. Vertex coordinates can theoretically have any coordinate, but fragments can't since they are bound by the resolution of your window. There will almost never be a one-on-one mapping between vertex coordinates and fragments, so the rasterizer has to determine in some way at what fragment/screen-coordinate each specific vertex will end up at.

Image of a triangle being rasterized in OpenGL

Here we see a grid of screen pixels where the center of each pixel contains a sample point that is used to determine if a pixel is covered by the triangle. The red sample points are covered by the triangle and a fragment will be generated for that covered pixel. Even though some parts of the triangle edges still enter certain screen pixels, the pixel's sample point is not covered by the inside of the triangle so this pixel won't be influenced by any fragment shader.

You can probably already figure out the origin of aliasing right now. The complete rendered version of the triangle would look like this on your screen:

Filled triangle as a result of rasterization in OpenGL

Due to the limited amount of screen pixels, some pixels will be rendered along an edge and some won't. The result is that we're rendering primitives with non-smooth edges giving rise to the jagged edges we've seen before.

What multisampling does is not use a single sampling point for determining coverage of the triangle, but use multiple sample points (guess where it got its name from). Instead of a single sample point at the center of each pixel we're going to place 4 subsamples in a general pattern and use those to determine pixel coverage. This does mean that the size of the color buffer is also increased by the number of subsamples we're using per pixel.

Multisampling in OpenGL

The left side of the image shows how we would normally determine the coverage of a triangle. This specific pixel won't run a fragment shader (and thus remains blank) since its sample point wasn't covered by the triangle. The right side of the image shows a multisampled version where each pixel contains 4 sample points. Here we can see that only 2 of the sample points cover the triangle.

The amount of sample points can be any number we'd like with more samples giving us better coverage precision.

This is where multisampling becomes interesting. We determined that 2 subsamples were covered by the triangle so the next step is to determine a color for this specific pixel. Our initial guess would be that we run the fragment shader for each covered subsample and later average the colors of each subsample per pixel. In this case we'd run the fragment shader twice on the interpolated vertex data at each subsample and store the resulting color in those sample points. This is (fortunately) not how it works, because this basically means we need to run a lot more fragment shaders than without multisampling, drastically reducing performance.

How MSAA really works is that the fragment shader is only run once per pixel (for each primitive) regardless of how many subsamples the triangle covers. The fragment shader is run with the vertex data interpolated to the center of the pixel and the resulting color is then stored inside each of the covered subsamples. Once the color buffer's subsamples are filled with all the colors of the primitives we've rendered, all these colors are then averaged per pixel resulting in a single color per pixel. Because only two of the 4 samples were covered in the previous image, the color of the pixel was averaged with the triangle's color and the color stored at the other 2 sample points (in this case: the clear color) resulting in a light blue-ish color.

The result is a color buffer where all the primitive edges now produce a smoother pattern. Let's see what multisampling looks like when we again determine the coverage of the earlier triangle:

Rasterization of triangle with multisampling in OpenGL

Here each pixel contains 4 subsamples (the irrelevant samples were hidden) where the blue subsamples are covered by the triangle and the gray sample points aren't. Within the inner region of the triangle all pixels will run the fragment shader once where its color output it is stored in all 4 subsamples. At the edges of the triangle not all subsamples will be covered so the result of the fragment shader is only stored at some subsamples. Based on the amount of subsamples covered, the resulting pixel color is determined by the triangle color and the other subsample's stored colors.

Basically, the more sample points are covered by the triangle, the more the eventual pixel color is that of the triangle. If we then fill the pixel colors just like we did earlier with the non-multisampled triangle we get the following image:

Rasterized triangle with multisampling in OpenGL

For each pixel, the less subsamples are part of the triangle, the less it takes the color of the triangle as you can see in the image. The hard edges of the triangle are now surrounded by colors slightly lighter than the actual edge color, which causes the edge to appear smooth when viewed from a distance.

Not only color values are affected by the multisampling, but also the depth and stencil test now make use of the multiple sample points. For depth testing, the vertex's depth value is interpolated to each subsample before running the depth test and for stencil testing we store stencil values per subsample, instead of per pixel. This does mean that the size of the depth and stencil buffer are now also increased by the amount of subsamples per pixel.

What we've discussed so far is a basic overview of how multisampled anti-aliasing works behind the scenes. The actual logic behind the rasterizer is a bit more complicated than we've discussed here, but you should be able to understand the concept and logic behind multisampled anti-aliasing now.

MSAA in OpenGL

If we want to use MSAA in OpenGL we need to use a color buffer that is able to store more than one color value per pixel (since multisampling requires us to store a color per sample point). We thus need a new type of buffer that can store a given amount of multisamples and this is called a multisample buffer.

Most windowing systems are able to provide us a multisample buffer instead of a default color buffer. GLFW also gives us this functionality and all we need to do is hint GLFW that we'd like to use a multisample buffer with N samples instead of a normal color buffer by calling glfwWindowHint before creating the window:

glfwWindowHint(GLFW_SAMPLES, 4);

When we now call glfwCreateWindow the rendering window is created, this time with a color buffer containing 4 subsamples per screen coordinate. GLFW also automatically creates a depth and stencil buffer with 4 subsamples per pixel. This does mean that the size of all the buffers is increased by 4.

Now that we asked GLFW for multisampled buffers we need to enable multisampling by calling glEnable and enabling GL_MULTISAMPLE. On most OpenGL drivers, multisampling is enabled by default so this call is then a bit redundant, but it's usually a good idea to enable it anyways. This way all OpenGL implementations have multisampling enabled.


Once the default framebuffer has multisampled buffer attachments, all we need to do to enable multisampling is just call glEnable and we're done. Because the actual multisampling algorithms are implemented in the rasterizer in your OpenGL drivers there's not much else we need to do. If we now were to render the green cube from the start of this tutorial we should see much smoother edges:

Image of a multisampled cube in OpenGL

This container does indeed look a lot smoother and the same will apply for any other object you're drawing in your scene. You can find the source code for this simple example here.

Off-screen MSAA

Because GLFW takes care of creating the multisampled buffers, enabling MSAA is quite easy. If we want to use our own framebuffers however, for some off-screen rendering, we have to generate the multisampled buffers ourselves; now we do need to take care of creating multisampled buffers.

There are two ways we can create multisampled buffers to act as attachments for framebuffers: texture attachments and renderbuffer attachments, similar to normal attachments like we've discussed in the framebuffers tutorial.

Multisampled texture attachments

To create a texture that supports storage of multiple sample points we use glTexImage2DMultisample instead of glTexImage2D that accepts GL_TEXTURE_2D_MULTISAPLE as its texture target:

glBindTexture(GL_TEXTURE_2D_MULTISAMPLE, tex);
glTexImage2DMultisample(GL_TEXTURE_2D_MULTISAMPLE, samples, GL_RGB, width, height, GL_TRUE);
glBindTexture(GL_TEXTURE_2D_MULTISAMPLE, 0);  

The second argument now sets the number of samples we'd like the texture to have. If the last argument is equal to GL_TRUE the image will use identical sample locations and the same number of subsamples for each texel.

To attach a multisampled texture to a framebuffer we use glFramebufferTexture2D, but this time with GL_TEXTURE_2D_MULTISAMPLE as the texture type:


The currently bound framebuffer now has a multisampled color buffer in the form of a texture image.

Multisampled renderbuffer objects

Like textures, creating a multisampled renderbuffer object isn't difficult. It is even quite easy since all we need to change is the call to glRenderbufferStorage that is now glRenderbufferStorageMultisample when we specify the (currently bound) renderbuffer's memory storage:

glRenderbufferStorageMultisample(GL_RENDERBUFFER, 4, GL_DEPTH24_STENCIL8, width, height);  

The one thing that changed here is the extra parameter after the renderbuffer target where we set the amount of samples we'd like to have which is 4 in this particular case.

Render to multisampled framebuffer

Rendering to a multisampled framebuffer object goes automatically. Whenever we draw anything while the framebuffer object is bound, the rasterizer will take care of all the multisample operations. We then end up with a multisampled color buffer and/or depth and stencil buffer. Because a multisampled buffer is a bit special we can't directly use their buffer images for other operations like sampling them in a shader.

A multisampled image contains much more information than a normal image so what we need to do is downscale or resolve the image. Resolving a multisampled framebuffer is generally done via glBlitFramebuffer that copies a region from one framebuffer to the other while also resolving any multisampled buffers.

glBlitFramebuffer transfers a given source region defined by 4 screen-space coordinates to a given target region also defined by 4 screen-space coordinates. You might remember from the framebuffers tutorial that if we bind to GL_FRAMEBUFFER we're binding to both the read and draw framebuffer targets. We could also bind to those targets individually by binding the framebuffers to GL_READ_FRAMEBUFFER and GL_DRAW_FRAMEBUFFER respectively. The glBlitFramebuffer function reads from those two targets to determine which is the source and which is the target framebuffer. We could then transfer the multisampled framebuffer output to the actual screen by blitting the image to the default framebuffer like so:

glBindFramebuffer(GL_READ_FRAMEBUFFER, multisampledFBO);
glBindFramebuffer(GL_DRAW_FRAMEBUFFER, 0);
glBlitFramebuffer(0, 0, width, height, 0, 0, width, height, GL_COLOR_BUFFER_BIT, GL_NEAREST); 

If we then were to render the application we'd get the same output as without a framebuffer: a lime-green cube that is displayed using MSAA and thus shows significantly less jagged edges:

Image of a multisampled cube in OpenGL

You can find the source code here.

But what if we wanted to use the texture result of a multisampled framebuffer to do stuff like post-processing? We can't directly use the multisampled texture(s) in the fragment shader. What we could do is blit the multisampled buffer(s) to a different FBO with a non-multisampled texture attachment. We then use this ordinary color attachment texture for post-processing, effectively post-processing an image rendered via multisampling. This does mean we have to generate a new FBO that acts solely as an intermediate framebuffer object to resolve the multisampled buffer into a normal 2D texture we can use in the fragment shader. This process looks a bit like this in pseudocode:

unsigned int msFBO = CreateFBOWithMultiSampledAttachments();
// then create another FBO with a normal texture color attachment
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, screenTexture, 0);
    // now resolve multisampled buffer(s) into intermediate FBO
    glBindFramebuffer(GL_READ_FRAMEBUFFER, msFBO);
    glBindFramebuffer(GL_DRAW_FRAMEBUFFER, intermediateFBO);
    glBlitFramebuffer(0, 0, width, height, 0, 0, width, height, GL_COLOR_BUFFER_BIT, GL_NEAREST);
    // now scene is stored as 2D texture image, so use that image for post-processing
    glBindFramebuffer(GL_FRAMEBUFFER, 0);
    glBindTexture(GL_TEXTURE_2D, screenTexture);

If we'd then implement this into the post-processing code of the framebuffers tutorial we're able to create all kinds of cool post-processing effects on a texture of a scene with (almost) no jagged edges. With the blur kernel filter applied it'll look something like this:

Image of post-processing on a scene drawn with MSAA in OpenGL Because the screen texture is a normal texture again with just a single sample point, some post-processing filters like edge-detection will introduce jagged edges again. To accommodate for this you could blur the texture afterwards or create your own anti-aliasing algorithm.

You can see that when we want to combine multisampling with off-screen rendering we'd need to take care of some extra details. All the details are worth the extra effort though since multisampling significantly boosts the visual quality of your scene. Do note that enabling multisampling can noticeably reduce performance of your application the more samples you use. As of this writing, using MSAA with 4 samples is commonly preferred.

Custom Anti-Aliasing algorithm

It is also possible to directly pass a multisampled texture image to the shaders instead of first resolving them. GLSL then gives us the option to sample the texture images per subsample so we can create our own anti-aliasing algorithms which is commonly done by large graphics applications.

To retrieve the color value per subsample you'd have to define the texture uniform sampler as a sampler2DMS instead of the usual sampler2D:

uniform sampler2DMS screenTextureMS;    

Using the texelFetch function it is then possible to retrieve the color value per sample:

vec4 colorSample = texelFetch(screenTextureMS, TexCoords, 3);  // 4th subsample

We won't go into the details of creating custom anti-aliasing techniques, but merely provided some pointers as to how you might implement a feature like this.