"People of Colour"

The term "people of colour" annoys me a bit, and I've just thought of a good analogy to explain why.

You see, I'm a person of colour too: in my case, pinkish with definite overtones of red, particularly after I've had my first glass of wine following a week or so of abstinence.

The analogy is this: there are those of a posh English persuasion who seem to genuinely believe that they speak English "without an accent". As if their mode of speech were the default, and all others are deviant. And yet everybody else would describe them as having, ipso facto, a posh English accent. Simple.

You see what I'm saying?

Do you reckon I'm overthinking this, or would you agree? Or is this in fact a well-known gripe and I'm just massively behind the curve? Or maybe you think it's useful to have a catch-all term for people who are not "white" in order to talk about various kinds of racist oppression? If so, what should it be?