From all the tests I've done, task / quest xp is not affected by LotD or XP Potions. Now I haven't tested alot out thoroughly, but I have done the Farm Clean Up series, Worgs, and Wereorcs, and the Charm of Lore fairly extensively with and without LotD and/or XP potions, and has appeared exactly the same. However the kills still make burning lotd worth it at the farm or for the Wereorcs and Worgs quests, at least at the lower levels.
Yther Ore.