This is indeed what has been worked on lately, so congrats to those who guessed right :P

As well as finding out that my edge slope functions weren't perfect. I tried, with moderate success, to make them more accurate, but they're still not perfect. So for now, I need to let it cool down. I decided I would make antialiasing 'good enough', then start working on the UI. Considering that there are other areas of the GPU that aren't perfect, like polygon clipping.

So here's a couple screenshots:

I picked cases where the result is quite visible. Antialiasing generally makes things look better, but whether it is that visible depends on what's being rendered.

Antialiasing may look like one of those minor details that are unimportant. But I consider it important to emulate, past the effect of making things look nicer: the way it works on the DS is very different from common antialiasing techniques. If you're into DS homebrew, you can't just turn on antialiasing to magically make things look better.

To begin with, antialiasing is applied to polygon edges based on their slopes, in order to make the edges look smoother. There is no antialiasing applied inside polygons or at intersections between two polygons.

Antialiasing also doesn't apply to translucent polygons. But actually, it's more complex than that -- the polygon fill rules depend on whether the individual pixel being rendered is translucent. So if a polygon has opaque and translucent pixels at its edges, the opaque parts will be antialiased and the translucent parts won't be.

More importantly, antialiasing interferes with wireframe polygons, lines and points. Wireframes are drawn as if they were filled polygons, ie. only the outer side gets antialiased. Lines are similarly antialiased on one side only. Points are said to disappear entirely.

Edge marking is also affected in that marked edges are made translucent when antialiasing is enabled. No idea whether it was intended, but it needs to be taken into account.

The real gold is how the hardware is designed to ensure antialiased edges are always rendered properly. The design is quite atypical and inspired from how the 2D GPU does blending.

If antialiased edges were immediately blended against the pixels underneath, it could cause visible glitches if another polygon is later rendered behind an antialiased polygon. So instead, per-pixel edge coverages are put aside for later: antialiasing is a separate rendering pass. It is the final pass, applied after edge marking and fog.

That final pass needs to know what the colors under the polygon edges are. Thus, the GPU keeps track of the last two colors rendered. When rendering an opaque polygon, the existing color is pushed down, and the new polygon color is written at the top. This is the same design that is used by the 2D GPU for its blending.

But if you render a polygon behind another, that polygon's pixels can be inserted behind the existing polygon, effectively replacing the existing bottom-most pixels.

The limitation is that this only works for the topmost edge pixels. If you draw two identical polygons A and B at the same position, with A on the top, A's edges are blended against B, and B isn't antialiased.

Things are funkier when translucency is involved. As far as I understand, translucent polygons don't push the existing topmost pixel down, but they blend with both topmost and bottom-most pixels. Translucent pixels that are behind the topmost pixel can still be blended with the bottom-most pixel in they come in front of it.

melonDS doesn't emulate this bit yet, but I will have to rework the renderer code at this point to make things nicer to work with.

Similarly, fog is applied to both topmost and bottom-most pixels, so that antialiased edges are still fogged properly. This part is emulated but at the expense of some duplicate code.

I'm not going to cover in detail how edge pixel coverages are calculated, because I'm still not quite sure. I haven't even gotten the edge functions perfect yet. This GPU isn't done being weird, heh.

But enough GPU work for now. There are still many other areas that need work. And I really want to put out a release with a nice UI.

As a side note: much of the research was initially done by Cydrak, and dasShiny was the first to attempt antialiasing support (albeit incomplete) (not counting DeSmuME's option to use OpenGL AA).

Wow, so only the Y-axis edges is anti-aliased on dasShiny and it doesn't even look jarring all-in-all. I do think Melon's screenshots are becoming a whole lot better, more consistent-looking than that of the OpenGL renderer though.

Gimzie says:Sep 1st 2017

This looks really cool! Looking forward to trying games with this feature.