Okay... so what is the new model? What happens when I draw a color glyph
with operator XOR and a red source?

The red source is ignored for color glyphs because they are used as the source.

Note that while with Google's (CBDT/CBLC) and Apple's (sbix) color font formats the red source will be ignored, with the Microsoft (COLR/CPAL) and Adobe's (SVG/CPAL) the font can have color imaging as well as using the current foreground color. So, the glyph image won't be just a source either. It will be a mix of source and mask... Right now those two formats are not implemented in FreeType though.