So it definitely isn't a direct measurement of all 3 values at every pixel position any more, it's having to calculate values....

Yes, though you could say you always had to calculate values (noise, etc.), but yes, if you wanted to fill in certain colors at the same full 19.6MP resolution, you would have to calculate them. You can also ignore that and have lower chroma resolution in more explicit way and the luminance detail over it with also some color range being more detailed oriented.