I think most people would make a distinction between words that have been adopted into mainstream English usage, and foreign language words that we use because there isn't an English word. Hajj isn't a generic word that can be used for any other purpose. It has a specific meaning, and it is inappropriate - even offensive - to use it in any other context. I would never "go on a hajj to Bletchley Park", for instance, but I would "go on a pilgrimage" to it. So while technically it is "a word that is used in the English language", it's very much a borderline case. Although it's legal in Scrabble, so maybe I'm wrong.