It's impossible to destroy the totem. When you try, you get a "this item cannot be destroyed" warning.
Then again, I've had weird things happen with characters that have been inactive forever. My druid isn't able to re-pick-up the sealform quest, and if I ever actually decide to play him again I'll need to ask a GM about it.