Swords are better for PvP too, actually (Resilience means Fist or Axe spec is quite bleh, plsu I'm pretty sure you can get Sword Spec procs off Shiv).
If not for Combat Potency, there's no reason why Sword Spec would be better in PvE. But there is, and the extra Energy makes the difference. Strictly speaking though, it's better to crit and deal more damage (damage bonus plus 15% plus 3% from Metagem) than to get another white attack in.