I failed it the first time to, when you start the quest with the one guy, you need to stand in the middle of the building the entire time and wait. If you go with a group they need to be there as well, otherwise the guys will despawn. Just abandon the quest and try again.
Remember: Center at all times right after accepting it.