Well I don't really know where to start, but here's what I'd do if I was in your shoes:
Go back to old...old instances like BRD or even the lower level stuff so you can see how instances are set up. Then move up to Hellfire Citadel, (which you can close to solo, bring a healer friend if you need to) and watch the make-up of the instance. Hell, stealth through them.
Here's the basic idea - there are lots of 'pulls' involving 3-5 mobs of lower lvl elites. Either a) all melee, b) melee with casters/healers, or c) both. Most groups will crowd control a caster (sap for you) and 'focus fire' (everyone attacks same target) the healer. Then move on to the next dps, yada yada, until you get to the CC'd mob. Some groups mark (most don't anymore), but the general kill order is Skull, X, Blue Square, Moon (if needed) and either pink diamond or yellow star for sap. There are also patrols (pats) that walk in a predetermined path. Usually a timed shot by the tank is needed to pull them to the group, and not aggro other pulls.
Then you get to a boss, in most cases they will have some sort of special ability, spell, or adds (extra mobs that spawn and kill you). Most bosses at lvl 80 content are pretty easy, and you can read the boss strategies at www.wowwiki.com or wowhead or even here.
You can do instances on regular or heroic mode, meaning the loot is better and the last boss always drops an epic, but they are harder to kill. Heroics also drop Medals of Heriosm (or whatever the name is, like the old BOJ) which you can spend in Dalaran for better-than-quest gear.
Your main goal as a rogue is dps. Some interrupts, some saps, but mainly DPS. We've got the easiest job in the book, just stand behind your target and beat away (hehehehe). For a newbie, maintain your position with the rest of the group, you don't want to body-pull any extra mobs. If you do...DONT run to the group. Just vanish, or if that's down...die. Wiping the group unneccasarily is cause for a group-kick.
If you've never, ever ran an instance, read on. If not, skip. The 'tank' is usally a pally or warrior (or DK now) with a ton of threat gen and dodge/parry, and their job is to grab all the aggro from the pulls, and keep the mobs attacking them. You need to Tricks of the Trade the tank before you start your dps, and make sure you stay on the tanks target (or the marked target if applicable). Some times a tank will announce they are LOSing, that means Line of Sight, and they are going to pull the targets back around a corner to avoid casters and/or pats, etc.
Why not try and run some lower stuff with your guild? If you are 80 I don't think you can lfg (look for group) the lower instances, but try advertising on the /trade channel. Most level 71's would enjoy having an 80 in the group, since it's going to cut down on the run time and you wouldn't need the gear. BUT - at lvl 80, you are missing out on a ton of gear from reg and more likely heroic instances, not to mention the badge gear you can buy.
Good luck!
EDIT: More info added
Edited, Feb 11th 2009 5:24pm by ThomasMagnum