inconsistent alpha values in multisampled buffer

I am creating a Windows bitmap from a OpenGL multisampled pbuffer with a transparent background. I am using the alpha channel to control transparency in the resulting bitmap. The problem is that the alpha component in the RGBA value is inconsistent with the color components.

For example, I have a 2x FSAA pbuffer. I reset the RGBA color to (0,0,0,0). I then draw a filled white triangle. A pixel near the edge of the triangle will have these RGBA values: (188,188,188,127). I would have expected the RGB components to be the same as the alpha component. As it is, when I draw the bitmap into my device context, some of the blended pixels have color components that have overflowed (RGB component value would be greater than 255 which produces incorrect results). I am using the Win32 API AlphaBlend to draw the bitmap. It uses the following formula (red component example used) for normalized [0-1] values:

Dst.Red = Src.Red + (1 - Src.Alpha) * Dst.Red

So if I have a white destination pixel, and I substitute in the unnormalized values, I get this:

Dst.Red = 188 + (255 - 127) * 255 / 255 = 316 (Overflow!)

In my example, if the RGB components were the same as the alpha value, then there would never be an overflow. So my question is, for white fragments, why are the resulting RGB components not the same as the alpha component?

To visualize what is happening, I created a transparent bitmap from a 32x FSAA image with a bunch of white triangles. The bitmap is drawn onto a device context with a white background. The entire image should be white but you can see the triangle outlines. This because of the color component overrun caused by the alpha value not matching the color component values:

Is there an OpenGL setting that will fix the mutlisampled alpha values to be computed in the same way that the RGB components are computed? Or is this a bug in the Nvidia OpenGL driver?

More information regarding the source of this problem. I tested my sample code on a Intel HD Graphics P4000 integrated video adapter and there is no problem. That is, for the edges of white triangles, the alpha value is always the same as the RGB component values. For example, if each RGB component is 127, the alpha value is also 127. Maybe I need to contact Nvidia about this issue...

For example, I have a 2x FSAA pbuffer. I reset the RGBA color to (0,0,0,0). I then draw a filled white triangle. A pixel near the edge of the triangle will have these RGBA values: (188,188,188,127).

What happens if you disable blending and leave MSAA enabled?
What happens if you disable MSAA and leave blending enabled?

Couple thoughts on causes: blending, MSAA, CSAA, coverage, filtering.

First, coverage + MSAA. If you aren't passing exactly the same start and end points for an edge in adjacent triangles, you have no reason to expect that the edge will be rasterized the same for both and thus set all of the samples in the boundary pixels (and even then, I don't know for sure that that guarantees position-invariance -- check the spec). That could lead to problems -- some subsamples are affected while others aren't. At a glance, it appears this may apply to your code.

Also, you're not doing anything with alpha-to-coverage are you? And I don't see where you're actually enabling blending and setting your blend function.

Second, blending. If you have the previous problem, this is just going to aggravate it since some subsamples will get the blend while others don't. Then we downsample to mix things and make it harder to figure out what happened.

Third, CSAA. Make sure you're using an honest-to-goodness-no-kidding pure MSAA mode. Not CSAA or an MSAA/CSAA mix, which seems to be common in the NVidia drivers nowadays. Could be a CSAA color quantization issue. For instance, here in the Linux NVidia drivers, for std system framebuffer formats, I see:

Fourth, filtering. If the driver is using a kernel filter which pulls in samples from adjacent pixels (ala Quincunx from the olden days), will give you some bleed-over between samples in different pixels, but this won't be the cause -- will just make it harder to infer what is going on with the individual subsamples.

You might render this to an MSAA texture instead and then use texelFetch to grab the individual colors assigned to the subsamples to see exactly what the values are pre-downsample.

Also might simplify your test case to using just 2 triangles so easier to debug/reproduce, and put this in a standalone GLUT test program and post so others can easily run it for cross-comparison.

Framebuffer sRGB was also mentioned -- good point and should be checked.

I tried using glDisable(GL_FRAMEBUFFER_SRGB) but it didn't fix the problem. Note that blending is not enabled. The problem happens when the edge of a triangle is downsampled from the 2x multisample buffer.

I registered as an Nvidia nvdeveloper member and filed a question/bug report with them. I'll see what they have to say.