Racism is being misused for it though and soon the definition of racism will change. It will not be derived from the word "race" but form the word xeno/xen.

Instead of using xenophobia, which few people understand, it's easier to just go with the flow and use racism in this case. However you want to disagree with you, I can only agree to that, but I will still use racism when I talk to a certain group to make myself easier to understand.

You are for example the exception to the rule and there are many more, but the majority just take it as true and stick to it. Taking it further, definitions of words is decided by the mass. Gay was a word that was a synonym to happy, but what about today? I kind say "I'm gay" to express happiness. Saying that would be me stating that I'm homosexual.