What you describe was the real reason I first downloaded a mini-map addon.
I think you could edit a file somewhere and move the placement of the button, but I have no idea where it is located, I have looked for it but gave up the search in favor of the minimap addon.
Even if you don't customize or move the minimap, the addon will give you control over the arrangement of all the buttons. (but you will customise that minimap, you will want to!)
I personally use Titan Panel and put as many icons on a hideable bar myself, that minimap gets a bit crowded.
Chinchilla and SexyMap are both nice addons, but beware... moving your minimap is the first step into becoming a ui-aholic. Once you have the ability to move everything, you won't be able to stop!