X and Y here ar fixed point with 8 bits of fraction, so >>8 to properly use them. When M7SEL bit 7 ($211a) is clear, further restrict X and Y to the range 0-1023. When bits 6 and 7 are both set, use the low 3 bits of X and Y to select from character zero for X or Y <0 or >1023.