They might be annoying to you, but, they come in really useful when you are trying to time using Totemic Recall (or whatever that spell is called). If you are looking to get back the maximum mana from your mana totem, you want to dismiss it in the last couple seconds of its life span. The timers make this easy.
I know that I did not answer your question, but I haven't figured out how to get rid of them myself. I am not all to hot on getting rid of them, but I don't know how.
Hope that is at least helpful.