Fashion vs FunctionalityFollow

So I bought a pair of bronze gauntlets yesterday and immediately took them off due to them looking absolutely horrid with the rest of my armor. So I was wondering if anyone else has run into this experience. I play a Red Mage type of character (GLA, PGL, THM, CON mix) so I tend to avoid heavy looking armors on my character. This usually leaves me with gear that is not specter for GLA at all. I know there is a penalty for it similar to using gear that you are not leveled up to. How severe is the penalty for that, does anyone know?