5 Answers
5

To take up another of your points, they're designed (in D&D) to serve very different roles. The yuan-ti are a race of creepy, sinister beings that serve as a rival to humanity. They build cities, create magic items and jewelry, have servants, and seek geopolitical power of some kind. They have a whole civilization going on, like elves or hobgoblins. But since they're freaky snake-men, they also embody the Other in a way that elves or hobgoblins don't. They look very different, they think weird reptilian thoughts, and whether or not you consider sex to be something that should come directly into an RPG, their presence raises unconscious questions about exactly how they breed (and with whom...).

Nagas are solitary boss monsters. They hang out at some distance from civilization, whether that distance is physical or social. They don't want the things that humanoid races want. Instead of serving as an uncomfortable parallel of humanistic society, they're clearly Big Monsters. Regardless of their alignments (if you use alignments), they embody the chaos and fear inherent in the wilderness - they're the opposite of civilization. You can draw some interesting rakshasa parallels there, if you like a little Hinduism in your roleplaying.

Alternately, you can say "they're all creepy snake dudes!" and just kill 'em.

It's not "version specific," it's "setting specific." In the Forgotten Realms there is a common origin for yuan-ti and nagas. From the FR wiki, and derived I believe from the FR supplement Serpent Kingdoms:

"Long before humans dominated the continent of Faerûn, the Creator Races ruled Toril. The reptilian Creator Race, the sarrukh, were foremost amongst these and built up great empires. They bred the first yuan-ti by magically experimenting with and breeding men with snakes. This way they also created naga, and through a similar process, lizardmen. They eventually fell from power and the resourceful yuan-ti rose up to claim their Creators' power vacuum, even while sustaining the empire of Mhairshaulk. Of the fragmented World Serpent deity that the sarrukh had worshiped, the yuan-ti venerated the strongest aspect, a cruel and despotic deity, Merrshaulk, who grew distant and aloof."

I don't believe there is an "official" answer beyond what you wish in your own campaign. Yuan Ti or serpent men seem to be a homage to the Robert E. Howard serpent men while nagas appear to have a more mythological basis. I would think about making naga the leaders of the serpent men---their spellcasting abilities, heightened intelligence, and not having any hands---makes it logical the yuan ti would do all the "grunt" work while the naga commanded them. Perhaps nagas are rarely spawned by serpent men sort of like a bee hive spawns a queen bee, with several nagas competing in combat until one survives to rule over the brood.

They are not really related. Nagas in Hinduism and Buddhism are related a to version of the Asian or Oriental Dragon. They can be good or evil spirits, in Chinese myth they would be at the same rank as Heavenly fairy or genie. One is even the protector of the Buddha. In Cambodia there is a legend of reptilian race that ruled a empire. These sound more like Yuan-ti.

DnD tends to take most mythological and fantasy creatures out of there original context and create a polyglot world with creatures that would not be together. Nagas if you follow there background would think of them as superior to Yuan-ti in every way and would no more rule them than any other intelligent creature. In fact you could see benevolent empires rule by Nagas as well as despotic ones.

As far as I can remember, D&D's nagas are human-headed snakes. They never have legs, arms or other features that together make something truly "humanoid".

There are similar (high ranking?) yuan-ti as well (again, as far as I can remember), but yuan-ti are more varied, each (subclass?) having a mixed set of human and snake features (limbs, body, tail, head, tongue, scales, skin etc.)

The question is interesting, though, if somewhat broad - I'd say the (3.5) core books leave the answer, that is, the development of the relationship of the two races up to the DM. (There might be setting-specific answers, of course, but the question had no pointer about this.)