Well,
First of all, if your group's average level is too low for the instance you made a bad start anyway. If I join a group for SM cathedral and there's a level 30 in the group (and no level 50 or such to make up for it) I ask the leader what he thinks he's doing. I'll politely inform both the lowbie and the leader that you cannot hope to complete SM cathedral with somebody so underlevelled for the place that he'll be 2-shot at every turn. If they insist on taking the lowbie (once again, without a higher level to make up for his level) I'll know they are plainly both bad and stubborn players, and that I don't want to group with them anyway. For heroics the same thing applies in gear. If you've got some all greenies healer you should mention that it won't work like that.
Second, if your group is levelled and geared up enough for the instance and the group set-up is alright (tank and healer are present), you should always be able to complete the instance, unless
A) You aren't levelled and geared enough anyway; this means you made a mistake at step 1. If you didn't adress the inability of a certain player you're at much at fault as the rest of the group. Unless you didn't know the instance, in which case you should have mentioned that to the other group members.
B) Your group setup is wrong; while you can generally do with for example a tank, 2 healers and 2 DPS, a group compromised of 4 (pure) tanks and 1 healer isn't going to do much good.
C) You're having 'one of those' groups where everybody is capable but you wipe a couple of times thanks to stupid accidents (bad fear, healer spilling stuff over his keyboard, etc). In these situations I usually stay, but someone else ends up leaving. I've completed plenty of runs like these though; usually the result is that you were just unlucky in the start but move trough the instance fine afterwards.
or D) You have a good old plain noob in your party who doesn't know what to do and is messing everything over. Try and watch him for a bit, then state the errors he's making (in a polite way) and tell him that he'll have to improve if you want to tackle the dungeon. From this point 2 things can happen:
-The noob will improve and you'll have a very succesfull run in which you even learned somebody something.
-The noob will start shouting at you, calling you an *sshole for criticizing the way he plays, etc. Unless the group leader kicks him, this is the point where you can safely say 'au revoir' because you can be 99.99% sure the group as it is is NEVER going to complete the instance. If you leave at this point, nobody should even dare to question your ettiquette since you were plainly right and the noob can be blamed for messing up everything. This is what you should tell people who do attack you on leaving. Ofcourse, you'll have to be 100% sure you ARE right. If you're telling a MM hunter he's being a noob for not using Bestial Wrath you have absolutely no right to speak at all.
To summarize this, it all comes down to the following 3 steps;
1) When you join a group, assure that the gear, specs, levels and group make-up are sufficient enough to do the instance. If you're not sure about the instance, you'll learn this on the go and can comment about it then; just make sure you mention that you don't know the instance.
2) You should at this point be able to complete the instance without much trouble. If you dó end up in a wipefest, observe what's going on. Is your group simply being unlucky, then stick around and try to keep the motivation up.
3) If you've observed the situation and you've noticed somebody is messing up something, politely adress them and try to improve their play. If after trying a bit he (for whatever reason) does not succeed in improving his play, you can either ask the leader for a replacement or safely leave.
At least, that's my ettiquette.