Unexpected GPU triangle rasterization output

I'm trying to render the geometry that is shown in the following screenshot (wireframe mode):

This geometry is issued from dynamic tessellation, with fixed tessellation factors. As you can see, all vertices nicely join up, the geometry is "watertight".

Now let's see what this geometry looks like when rendered with a specific camera.

In filled triangles mode:

In wireframe mode:

Backface culling is disabled in both cases.

What a surprise! The GPU rasterizes more fragments/pixels in wireframe mode! It doesn't join up everything together when in filled triangle mode. In other words, filled geometry != wireframe fragments + some more fragments. How come?!
Is this expected? I don't see any bug on my side so far.
I'm on nVidia GTS 450 hardware and 320.18 drivers.

It's important to remember what GL is supposed to do when it comes to triangle raserization:

For rasterizing a triangle, a pixel gets filled if the center of the pixel is within the interior of the triangle.
For those pixels whose center is on the boundary of a triangle, the rasterizer has to work so that if two triangles share an edge, only one of those triangles gets rasterized.

For lines, the rasterizer draws a line of the thickness specified by glLineWidth between the two pixels that contain the two end points.

Now if most of the triangles are sub-pixel in size, it is quite likely that lines will rasterize more.

However, the picture given for the triangle rasterization is little worriesome. Indeed the actual mesh is connected, so its rasterization should be too. Do you have anti-aliasing enabled? That could explain why it is not connected.

It's important to remember what GL is supposed to do when it comes to triangle raserization:

For rasterizing a triangle, a pixel gets filled if the center of the pixel is within the interior of the triangle

Does this mean most fragments don't actually get rendered in my situation?
This rule makes sense but sounds a bit odd to me. Afterall, fragments at triangle endpoints usually do not lie in the triangle! So based on this rule, they should only rarely get rendered. That can't be true...
Is there a way to render fragments if the fragment intersects the triangle, and not if the fragment's center lies in the triangle?

[/LIST]
Does this mean most fragments don't actually get rendered in my situation?

When a group of triangles cover a fragment, the fragment's centre will lie within one of the triangles, and thus be rendered. OpenGL's rasterisation rules require that, in this situation, the fragment is rendered exactly once; otherwise blending wouldn't work correctly.

Originally Posted by fred_em

[/LIST]This rule makes sense but sounds a bit odd to me. Afterall, fragments at triangle endpoints usually do not lie in the triangle! So based on this rule, they should only rarely get rendered.

"Rarely" is overstating it. At any interior vertex, the fragment will lie within one of the triangles. The probability of it lying within a specific triangle is inversely proportional to the number of triangles surrounding the vertex. For a vertex at a silhouette edge, as the mesh resolution increases the angle between edges tends to 180 degrees and the probability of the fragment containing that vertex being rendered tends to 1/2.

Originally Posted by fred_em

[/LIST]Is there a way to render fragments if the fragment intersects the triangle, and not if the fragment's center lies in the triangle?

If multi-sampling is enabled, the fragment shader is run for each fragment which intersects the primitive. Thus, for fragments which intersect shared edges, the shader will be run multiple times. However, the result of each invocation will only affect the samples covered by the primitive being rendered; it cannot modify samples which lie outside of the primitive (if gl_SampleMask is assigned to, the value is AND-ed with the actual coverage mask).

As for what your original problem is, it would help if you provided more detailed images. My first suspicion is that you're only generating half as many triangles as you should be. If for every shared edge you render exactly one of the triangles sharing that edge, that won't affect the wireframe result but will affect the filled result.

Does this mean most fragments don't actually get rendered in my situation?

When a group of triangles cover a fragment, the fragment's centre will lie within one of the triangles, and thus be rendered. OpenGL's rasterisation rules require that, in this situation, the fragment is rendered exactly once; otherwise blending wouldn't work correctly.

Have a look at the following image:

Squares represent fragments (and associated screen pixels) that intersect a single triangle primitive (in red). Green squares are fragments which center does not lie in the triangle. Is the GPU supposed to raster both white and green fragments, or just green fragments? (I hope the answer is both, here).

"Rarely" is overstating it. At any interior vertex, the fragment will lie within one of the triangles. The probability of it lying within a specific triangle is inversely proportional to the number of triangles surrounding the vertex. For a vertex at a silhouette edge, as the mesh resolution increases the angle between edges tends to 180 degrees and the probability of the fragment containing that vertex being rendered tends to 1/2.

As for what your original problem is, it would help if you provided more detailed images. My first suspicion is that you're only generating half as many triangles as you should be. If for every shared edge you render exactly one of the triangles sharing that edge, that won't affect the wireframe result but will affect the filled result.

The problem I have here is that my rasterization is non-continuous. It is not a problem of shared edges or vertices, or something having to do with interior vertices.
My geometry is a series of 6 GL_PATCHES, they are perfectly connected, and the tessellation factors is 5x5 for each of them.

Squares represent fragments (and associated screen pixels) that intersect a single triangle primitive (in red). Green squares are fragments which center does not lie in the triangle. Is the GPU supposed to raster both white and green fragments, or just green fragments?

Just the white fragments, i.e. the ones whose centre lies inside the triangle.

Just the white fragments, i.e. the ones whose centre lies inside the triangle.

Ouch. I believe you, but I am just very surprised. I thought that, say, one-pixel sized triangles would always get rendered as a single fragment, but according to what you say, very often, they won't get rendered at all.
Is there a way I can force all fragments to get rendered? I do need the GPU to enter the fragment shader for all white and green fragments.
Not in front of my program right now but... if I enable antialiasing (GL_POLYGON_SMOOTH, hint=gl_nicest), will this work?

Originally Posted by GClements

If multi-sampling is enabled, the fragment shader is run for each fragment which intersects the primitive

Will multisampling really guarantee that both white and green fragments will be processed?
Using 4x multisampling, looking at the bottom row of fragments in my attached image, the first fragment from the left (which is green), will be treated as 4 samples. But the center point of all 4 samples all lie outside the triangle so... how can multisampling help me here?

Ouch. I believe you, but I am just very surprised. I thought that, say, one-pixel sized triangles would always get rendered as a single fragment, but according to what you say, very often, they won't get rendered at all.

If it worked the other way (i.e. all fragments intersecting a triangle were processed), fragments which intersect the edge between two triangles or the common vertex of any number of triangles would be rendered multiple times. That would produce the wrong result when using blending, glLogicOp(GL_XOR), etc. And it can't adjust the behaviour according to whether or not the edge is a boundary or interior edge because that isn't generally known at the point that each triangle is rendered.

Originally Posted by fred_em

Is there a way I can force all fragments to get rendered? I do need the GPU to enter the fragment shader for all white and green fragments.

Originally Posted by fred_em

if I enable antialiasing (GL_POLYGON_SMOOTH, hint=gl_nicest), will this work?

I believe so. Note that, if the colour buffer is fixed-point or floating-point (i.e. not integer), the fragment colour will automatically have its alpha value scaled by the coverage factor; you can't control this from the fragment shader.

Originally Posted by fred_em

Will multisampling really guarantee that both white and green fragments will be processed?

AFAICT, if none of the sample locations intersect the triangle, it's unspecified whether the fragment shader will be executed.(even if it is executed, the coverage mask will be empty and the result won't contribute to the final pixel colour).

If it worked the other way (i.e. all fragments intersecting a triangle were processed), fragments which intersect the edge between two triangles or the common vertex of any number of triangles would be rendered multiple times. That would produce the wrong result when using blending, glLogicOp(GL_XOR), etc. And it can't adjust the behaviour according to whether or not the edge is a boundary or interior edge because that isn't generally known at the point that each triangle is rendered.

This makes sense.

Originally Posted by GClements

I believe so. Note that, if the colour buffer is fixed-point or floating-point (i.e. not integer), the fragment colour will automatically have its alpha value scaled by the coverage factor; you can't control this from the fragment shader.

I tried with a regular GL_RGBA pixel format and get the following:

So it's better, but it isn't perfect. I verified the values in the depth buffer and the missing pixels haven't been affected (unfortunately). Tried GL_NICEST and FASTEST, same result.