3 Answers
3

TL;DR: Charisma is often connected to appearance, but has no inherent association with any physical qualities--and definitely does not imply comeliness. The repeated references to Charisma as a physical attractiveness score are incomplete shorthand, referencing the most common use of Charisma for player characters.

But first, I'd like to point out that this entry describes Str, Dex, and Con as "physical abilities" and the other three as "the remaining abilities" (inotherplaces, Int/Wis/Cha are called "mental ability scores").

Charisma: Reflects the creature's sense of self and ability to influence other creatures for good or for ill (a particularly scary creature often has a high Charisma score). A creature can have a very low Intelligence score and still be Charismatic. A Charisma score of 3 or lower indicates a creature that is barely sentient [aside: I think they mean "sapient"]. A score of 4 to 6 represents Charisma equivalent to that of a strikingly sullen, crass, or retiring human. A score of 16 to 18 reflects an unusually strong presence and force of personality. A score of 20 or higher represents superhuman Charisma. [3.5 MM1, 298]

From this, it is clear that Charisma is not a beauty stat: an exceptionally ugly character might have that appearance represented by a high Charisma too. However, if we read further into this paragraph we find that Charisma is not technically a measure of physical appearance at all.

Strictly speaking, Charisma represents a creature's interpersonal skills. That is, a creature with high Charisma knows how to use its beautiful or hideous appearance to influence others. A terribly ugly or astonishingly pretty creature that doesn't have control over how its features influence others has low Charisma, and a man with an ordinary face who is masterful at influencing people has high Charisma. Remember, Charisma is the base stat for both Intimidation and Diplomacy.

ASIDE: The D&D Fourth Edition description of Charisma says the same thing in many fewer words.

Charisma is a mental score in this definition, as shown in the table where it goes up along with Intelligence and Wisdom.

Now, it's clear from your quote that Charisma can affect a physical characteristic of your character (looks), but it doesn't have to. You can have an ugly character with a respectable Charisma, or an attractive one with a low Charisma. Because of that, it's not considered a physical score.

The important thing to remember is that the effects of Charisma are subjective to the target, which is also separate from other possible race/culture modifiers. It combines aspects of personal magnetism with certain aspects of physical attractiveness to achieve a certain result.

You could have someone who is physically attractive to most other species (say, elves), but has a horrible personality, or their perfections take a creepy or unattractive bent (a little too symmetrical, huge Elfquest style ears, etc).

I remind my players when I start a game about this, and that its not a toss away stat. An elf player in my game reiterates it in his role playing, making various racial comments about others now and then, or extremely arrogant comments to emphasize it.