While it's true that unarmed skill affects Shield Slam only when you're disarmed, I think it's still something worth leveling up. Get yourself some int buffs and a fist weapon and go kill some stuff for 30-40 minutes to max it out.
I know there aren't that many situations where you'll find yourself disarmed, but it's nice to know that you can still generate some decent threat when you are. Disarm effect typically only last 3-6 seconds, during which time you can TC, Shield Slam, Shield Bash, and/or Demo Shout to keep your threat up.
Well worth it IMO.