If you're starting as a Conjurer and want the ambiance of the Conjurer storyline & the closeness of their guild, Gridania is your best pick. If you're starting as a Thaumaturge and want to start close to the Thaumaturge guild, Ul'dah is where you want to start. ^.^b

You can buy stuff for mages in any of the three cities, but it should also be noted that you can only get guild marks for Conjurer from Gridania-based leves and Thaumaturge guild marks for Ul'dah-based leves.

Go with the one you like the most. THM is in Ul'dah, CON is in Gridania and Limsa will have one eventually. No matter where you start you can hang out in whichever city you want too. The main difference is their starting story line and their grand champion story line. Enjoy the game.