__________________
"You'd be surprised how many people violate this simple principle every day of their lives and try to fit square pegs into round holes, ignoring the clear reality that Things Are As They Are." -Benjamin Hoff, The Tao of Pooh

The first one requires that you see the GUID of one of the mobs either through hovering over one or through the combat log, then using that to construct a tooltip and checking if it contains quest text and building a table using the mob name for the key.

That way if you see a nameplate with a name that's in the table then you can be reasonably confident it's a quest objective.

The other way would be to just take the name out of the nameplate and search through your quest log's quest objectives for a match, which might be more reliable but probably more intensive.

There's an interface option in the game under Names, in a dropdown menu under NPC Names you can select Quest NPCs to make their names always appear, which makes them more noticeable.

edit: Actually the second option wouldn't work, you'd pretty much have to do tooltip scanning for this. The mob's name is not necessarily mentioned in the quest text at all.

__________________
"You'd be surprised how many people violate this simple principle every day of their lives and try to fit square pegs into round holes, ignoring the clear reality that Things Are As They Are." -Benjamin Hoff, The Tao of Pooh

This isn't exactly what you want, but there is a built-in way to distinguish between normal NPC's and quest NPC's. In the Interface options, under Names, there is an option called NPC Names. In the dropdown, select Quest NPCs. If you see a name displayed above their head, you'll now know that they are related in some way to one of your quests.

Like Talyrius says, you can use Blizzard's default UI to display names of mobs associated with quests. This works well from a distance, but if your nameplates change at closer range you lose that aspect. This is how I play anyway. You might also investigate the addons Wholly and Grail. Grail is a quest database, and Wholly provides UI for that database. Part of the changes Wholly makes is to modify the tooltips for NPCs to list whether they need to be killed to start a quest, or drop an item that starts a quest. This will not tell you that you need to kill X of these for a quest, but it is a little closer to what I think you want.