Having a LOT of haste can make a difference. Spells/abilities like Bloodlust/Heroism/Icy Veins that increase your haste by X% (20% for instance) makes a more noticeable affect on your casting speed than having, say 100 haste from gear. Always stack spellpower first (it will be the primary stat on almost all of your gear anyway...) but if you want to ignore crit for the most part you can do that (although again, it will come on gear later with haste and spell power!).
So yes and no. Haste makes a somewhat noticeable difference if you watch your cast time to the fraction of the second. For instance, my 2.5 second casts now cast for about 2.1 seconds with about 15% haste, which is a fairly sizeable amount of haste stacking at level 80 (specifically gearing for it). Now, with bloodlust+icyveins that will cut down to be about 1.5 second casts and so forth, which, I'm sure as you can tell, makes a seeminly bigger difference (almost cutting cast time in half).
I hope that helps.