If you're starting as a Conjurer and want the ambiance of the Conjurer storyline & the closeness of their guild, Gridania is your best pick. If you're starting as a Thaumaturge and want to start close to the Thaumaturge guild, Ul'dah is where you want to start. ^.^b
You can buy stuff for mages in any of the three cities, but it should also be noted that you can only get guild marks for Conjurer from Gridania-based leves and Thaumaturge guild marks for Ul'dah-based leves.