Either way you shouldn't be getting hit physically by any monster, so your armor means precious little. What can hurt you is AOE attacks which are oftentimes magic (thus ignoring armor), which means what matters is your health value (and by association, stamina); raiding often requires a minimum amount of health to be effectively survivable in the encounters faced, which sometimes requires you give up some of your caster stats in favor of a little more stamina (though I don't know if this is still the case in a post 3.0 world, the gear you get by the time you're 80 may be enough health anyway).
Either way it's not a terrible concern while leveling up, just improve the caster stats and the health will generally follow suit along with you.