Prexus Server Meltdown

Bad news for players on Prexus. The server broke down last night and all characters are being restored from backups. If you played yesterday, you will lose whatever items and experience you gained. Here is Alan's post on it from the Developer's Corner. ----------- Last night at about 1:00 in the morning Prexus had a severe breakdown. Our Operations team has been working to get it restored and back into service. Our hope is that we'll have it back up and running some time this evening. That's the bad news. The worse news is that we will have to recreate all of the characters on the server from backups that are perhaps more than 24 hours old. Anything those characters have earned in the last day or so will be lost. Unfortunately, we will not be able to restore items or experience lost to the roll back. There is some potential good news, though. If you have deleted a character in the last year or so, that character may return. So if you see a character that you deleted a few months ago on your character select screen, that's why. Important Note: The restoration of these characters may leave you with more than eight characters on Prexus. In cases like this, the character select screen will list your characters in alphabetical order, and only show you the first eight of those characters. This may make some of your characters appear to be missing. They are not missing. All you need to do is delete one or more of the visible characters in order to see the ones that are not shown. We are extremely sorry that this has happened. We are doing everything we can to restore the server. Alan

Comments

« Previous 1 2
Post Comment
woot!
# Jun 24 2003 at 5:22 AM Rating: Default
Well, all i can say woot! I just discovered 5 characters i deleted months ago and they had a combibned value of over 200,000 pp in items and cash on them ! WTG Sony!
RE: woot!
# Jun 27 2003 at 8:32 AM Rating: Default
awesome :) glad you had a positive side in the crash while others had negitives :) grats.
play on?
# Jun 21 2003 at 2:33 PM Rating: Default
People have the right to be mad if they want to and if those who dont like it should not worry about it as matter as ignore it.If you were to buy something and it says like dare explore this contenent and all that stuff to lure you .. youll think its good and so you pay and play and even tho it may be theirs and w/e eula i dont care statement its your money.It would be nice if they could do something in return since sony is known as "global" in japan but they said sorry and that was it? yea more like playin with a persons emotion.Hey i just say quit and move on cause prexus has a record of crashes..and for my last 2 cents most of you ppl that say "get over it"?...do you even play on prexus as well as take you "entertainment time /w events memories stuff etc.." into it 99% 1% is the excuse.
RE: play on?
# Jun 25 2003 at 8:51 AM Rating: Decent
lol....what???
RE: play on?
# Jun 23 2003 at 2:36 PM Rating: Default
I have to appologize. But I just read your post twice, and I have to ask. Is that english? Please learn how to phrase a complete sentence before posting on one of these boards. Illegible jargon like that, does absolutely nothing to convey your opinion. It just makes you look like an idiot.
RE: play on?
# Jun 26 2003 at 8:01 PM Rating: Default
Well if you had a good education maybe you would understand but not all ppl graduated
RE: play on?
# Jun 27 2003 at 2:35 PM Rating: Default
ROFLMAO! If you had a decent education you would not have tried defending that with a lame flame. LOL. Go back to reading your comic books, and dreaming about being an astronaut.

BTW, to the original poster, thanks for the more articulated version of your post. I understood that one, and agree with you. I was not trying to flame you with my first post. I litterally did not understand half of your post.

Thanks
RE: play on?
# Jun 27 2003 at 5:04 PM Rating: Default
I'm the original poster of play on, and it was just on server crashing and thats it ill never post here again on my opions if this what pops up. sorry bye
RE: play on?
# Jun 27 2003 at 5:01 PM Rating: Default
Ok now what does a decent education have to do with defending my post and 2 i was talking about how ppl had the right to be hurt on how they lost there chracters and 3 i didnt insult anyone in that post None what so ever! so if you didnt like the post DONT READ IT NO ONES MAKING YOU TO READ IT,Good Day.
RE: play on?
# Jun 26 2003 at 8:16 PM Rating: Default
Lol my fault for the last one, what im just saying is that since sony is known well to me at lease to be "big and reliable" (been with them for some time)and not what happens is not their fault (not sure on that) just saying that ppl that are upset have the right to be upset eula or not for the time and money spent playin it etc whatever there reason may be.If you dont understand the rest oh well...
It's a game people..
# Jun 16 2003 at 2:57 PM Rating: Decent
Mommy! Mommy! Someone took my toys away from me! WAAAH!

Please people.. try to act your ages! So Prexus died, and you lost a little bit of stuff.. You still have your characters.. better yet, you're still alive! EQ is a game. a G A M E. For those of you who don't understand that.. that means ITS NOT REAL.

If you go this nuts over a game, how do you survive real life?
Most important question WHO's NEXT!!
# Jun 16 2003 at 2:15 PM Rating: Default
Do Sony Network Engineers have no pride? You have to be kidding me that they are admiting in the open public that a server "crashed", let alone they cant do anything better that "24 hour" before crash recovery?????

Are you runing the games on 486 systems with rewired christmas lights for network cables and a 100 meg paralell port Zip drive for back up???

No self respecting Network engineer will admit that a server "crashed", their are always techinical difficult circumstance surrounding software or hardware, but a "crash' is bogus general statement. IF THE FANS ALL STOPPED IN THE SERVER AND THE PROCESSOR OVERHEATED TO TUNE OF A SMALL SUN, AND MEMORY CHiPS MELDED TOGTHER, its still recoverable, EVER HEARD OF A HOT STANDBY SERVER??

Swap the "in need of repair" server with the hot standby, outtage time, 2-4 hours max, WHY you ask?

On the fly, out of band, fiberchannel based up to the minute backup system, THATS WHY!!

SOE/VEREAnt if ya need some contact for back up systems or disaster recovery let me know be glad to help, ever heard of SUNGUARD, COMMVAULT, VERITAS, COMPUTER ASSOCIATES??? hmmm

An Out of band fiber channel backbone to an opticial or large raid array(not your home version, big iron DASD for those who know what im talking about) that has week capacity, rotated out to robot tape libray every week.

This is no brainer stuff, How can you possible state what you have, this is unacceptable.

Im not sure what 12 year old paper certified engineers you have, but you need to tell them
to put down the lincoln logs and mp3 players and do some research or call a consultant team on how to manage a network:

RULE NUMBER ONE RUNNING A NETWORK:
YOUR ONLY AS GOOD AS YOUR LAST BACKUP, PERIOD!

Im also not sure what hardware platform your running on, and I know folks have talked about clusters which are ok, but still have to high a failure rate in endurance, database based apps like EQ is, but CALL IBM AND GET SOME OF THOSE
CMOS Mainframe boxes and drop the LAN party mentality.

This is shoddy network management, thoroughly unacceptable, and utterly confidence blowing.

I say again based on their no pride in systems and bad management, WHO"S SERVERS ARE NEXT?


Edited, Mon Jun 16 14:38:09 2003
RE: Most important question WHO's NEXT!!
# Jun 16 2003 at 8:36 PM Rating: Default
I am amazed at the ability of some people to make themselves look stupid.
RE: Most important question WHO's NEXT!!
# Jun 17 2003 at 11:50 AM Rating: Default
Yah! It never ceases to amaze me. Hey Mr. Systems Analyst. I can't believe you are talking DASD. Do you enjoy your EQ experience for less than $13 a month? Why don't you give your buddy at EMC a call and see what the price tag on enough DASD to backup all the EQ servers would be. We are not talking $129 for a 80Gig Maxtor here bud. You are talking min. hundreds of thousands of dollars. More than likely into the millions, just for the DASD, not to mention all the supporting hardware.

Buck up! You lost a few hours of playing time on a game. The support staff lost a lot of sleep and personal time in RL. You pay for a service, they are doing the best they can. Deal with it.
RE: Most important question WHO's NEXT!!
# Jun 17 2003 at 5:50 PM Rating: Decent
First, I dont think you understood the focus of my post, the backup system should have been in Place all along, online, backing up in real time. The technology has been around for sometime.

Second, might want to get your calculater out and multiply 13 bucks times couple hundred thousand account... I dont think SOE is hurting for money, if thats concern.

Your right as well as most folks in saying, if you dont care about the lost time, and they dont care about the lost time, then hey why should I. I dont even play on that server.

My surprise was they admitted a crash and to having such a poor back up system.

to each their own, live long and prosper.
RE: Most important question WHO's NEXT!!
# Jun 19 2003 at 2:23 PM Rating: Default
Its better they admit one of their servers crashed, than to completely lie about it. Just deal with it.
RE: Most important question WHO's NEXT!!
# Jun 19 2003 at 1:04 PM Rating: Decent
I too work in the technology department, and am well aware of the solutions available. But what you are proposing does not seem feasible to me, even as a player of the game.

You are talking about a huge amount of data, and therefore a tremendous amount of hardware. There is no need to get out the ol' calculator and figure out what $13 times a couple hundred thousand accounts is. Fact of the matter is, expanding operating expenses ALWAYS get passed on to the consumer. You want your million dollar backup, in the off chance that this is gonna happen again, to your specific server? Then shell over the cash. Or would you rather save a couple bucks a month, and take a chance? Worst case looks like you lose a days worth of play, if you did happen to play on the day it happened.

That is my point. Since I have 3 accounts in my name, I would rather save the cash X 3. These are not financial transactions taking place here, where loss of a single transaction can result in tremendous consequences. It is a game. Have you ever had a hard drive crash? You lose your saved games back to your last backup. That is what has happened here. Frankly, I would be pleased to know that only 1 day was lost. I would have expected more.

to sum things up...
# Jun 16 2003 at 3:50 AM Rating: Default
basically, i think that what people have already said covers this, but i'll insert my own 2cp here anyways...

EQ is a game, and losses like this in a game are disappointing to be sure, but that does not change the fact that they are not REAL.

When you feel that a minor, or even a major loss in a game is so significant that you should be reimbursed with REAL things, such as free play time, that means you have lost sight of the fact that EQ is a game, and it is meant to be played for entertainment purposes only. I have always been amazed at the number of people who seem to lack the ability to sit back and remember that there is a real world out there, and no matter how badly things go on an online game, the real world is more important. Sure it is a shame for those who have lost progress in this crash, but even if you lose a million platinum on EQ, you haven't lost a single dollar of RL money.

As for those who want in-game reimbursements, that matter has already been discussed, and just look to those reasonable people who have detailed the reasons such reimbursements are impractical for my stance. I agree that it is a shame that reimbursements are not possible, but i do not blame SOE for being reasonable.
Lost a guy
# Jun 15 2003 at 6:26 PM Rating: Default
Lost a level 10 Warrior who I had just given 70k to for items... Created him the day of crash... lost money and the guy... GMs say no record of him... but i had created him...
RE: Lost a guy
# Jun 16 2003 at 6:02 PM Rating: Decent
how did you lose the money? it should be back on whatever character you gave it to him from..... they didn't just roll back one character, they rolled back ALL of them =P

maybe i'm off on they way the rollbacks work, but i dont think so...
Not on the Prexus Server, but...
# Jun 15 2003 at 11:30 AM Rating: Decent
30 posts
Considering the amount of activity on any given server, wouldn't you think it would be wise for Verant/SOE to do backups on their servers at various points in the day? I would think if they were to revolve backups once every 8 hours it would be a lot less stressful to people than a 24 hour gap. I also find it odd that they haven't streamlined this problem to the point where they wouldn't have glitches like old characters popping up from a year before. Why would this information be retained on a snapshot from just a day earlier? I can understand they may want to retain some history of your deleted characters in case you have a nasty habit of scamming people and deleting characters, but beyond a month or two I can't see this information being useful.

Just my 2cp...
Wenasa Masterchef
46 Half Elf Ranger of Karana
Anyonae Ro
Do the math
# Jun 14 2003 at 9:53 AM Rating: Decent
What's EQ cost these days? (I can hardly remember, I never see the bill.) Like, 12.99 a month? OK. Divide your 12.99 by 30 to figure out how much you pay for eq per day. Take Sony to Small Claims court for your 43.3 cents in lost play time. You'll lose anyway.
Spare a though
# Jun 14 2003 at 7:09 AM Rating: Decent
Spare a thought for those that stayed at work around the clock to fix it. Imagine if they just said ... its 1730 I'm off home no more looking a broken servers til 9AM tommorow ?

Mixed Blessing
# Jun 13 2003 at 3:14 PM Rating: Decent
*
76 posts
LOL...this is way too funny. While most people are whining and crying at the roll back, I have a friend on that server that is just jumping for joy.

You see, I had a friend on that server that just recently got his butt scammed to hell. He lost out 40K worth of plat and 30K worth of gear on a deal that never happened. (Don't ask me about the scam. Just rest assured that he was being really stupid and way too trusting for his own good.)

Anyway, he was so pissed off at the scammed that he had considered retiring his character. Well, now that the server crashed and is about to roll back, he's one happy bugger. Granted that he will probably lose some exp and maybe a few plat that he made from selling; but it's a small price to pay for the stuff he got back.
#Anonymous, Posted: Jun 13 2003 at 2:52 PM, Rating: Sub-Default, (Expand Post) I'm from Prexus Server and I'm butt-hurt. Settle down people...They aren't going to give you all special items or money because of a problem that was unpreventable.
#Anonymous, Posted: Jun 13 2003 at 1:39 PM, Rating: Sub-Default, (Expand Post) Ho3w many times has prexus ummmmm my guess is 10 I think its time for a split.
#Anonymous, Posted: Jun 13 2003 at 12:29 PM, Rating: Sub-Default, (Expand Post) I don't understand this. You people seem to think it was only a days worth of stuff that was lost, after that they go to a previous record. why is it then that a character, that has been deleted for about a YEAR sudenly appeared and removed my main? I did get my main back, however you can't tell me that they only make backups every year or so.
#REDACTED, Posted: Jun 13 2003 at 2:30 PM, Rating: Sub-Default, (Expand Post) There are backups done every day... but even though a character was deleted there may still be a backup laying in your accoutns folder on the server. They wil be doign a blanket restoration, no real thought process in it, so if there is a backup, it gets restored, thats why there may be more than 8 characters appearing on your select screen.
#REDACTED, Posted: Jun 13 2003 at 12:08 PM, Rating: Sub-Default, (Expand Post) If the backup failed, And they have to restore from a older backup. Not that i would enjoy that i just think with it being Friday the 13th... Im very superstisious(sp bad things are bound to happen to good people. Its a shame really.
what matters?
# Jun 13 2003 at 9:06 AM Rating: Decent
Scholar
Avatar
**
551 posts
how many players do you think were on that day?

what if they had no way of restoring characters but had the logs of every person and what they did that day?

I think people are more upset that they MIGHT have a chance to get back what they did that day... but there is a snowball's chance in HELL that Sony is going to do ANY logs for a single person when they'd have to do it for probably THOUSANDS!

could ALL the information have been lost.. YES... does it matter.. NO!
____________________________
Pain Lord Darknyss Grimscale
92 Iksar Crusader of the Greenmist
Reimbursements?
# Jun 13 2003 at 8:52 AM Rating: Decent
I wonder if they're going to find any way to repay those who lost stuff. Or even just give an across the board bonus to all players on that server for the lost exp/items. They really should do something. Even if it's some coin or a free month of service for players on that server(they can afford that I'm sure).
RE: Reimbursements?
# Jun 13 2003 at 11:36 AM Rating: Decent
Hmm..reimburse you for items they own get lost? remeber, everything...EVERYTHING on your character belongs to Sony/verant..so they can remove items if they wish...so you "losing" an item they own because of the crash does not deserve a reimbursement. It's like saying "hey I lost your car but since I had it..can you give me a new one". As always, if you don't like it...quit....and have a nice day!


Go on, hate me, I kinda like it.

55 Troll Shaman
7th Hammer
RE: Reimbursements?
# Jun 13 2003 at 1:07 PM Rating: Default
That's the most absurd argument I think I'll ever hear. Why did you even waste your time, unless it was just to be contrary. Glad you're not on Rodcet Nife so I don't gotta listen to you in game
#REDACTED, Posted: Jun 13 2003 at 2:28 PM, Rating: Sub-Default, (Expand Post) May be absurd.. but he is right... Ever read the EULA? Heh, don't whine...
life goes on!
# Jun 13 2003 at 7:57 AM Rating: Default
ITS FRIDAY THE 13th, BAD STUFF HAPPENS!!!
HA HA
# Jun 13 2003 at 7:24 AM Rating: Default
I mostly find myself reading the forums to laugh at all the people complaining about "this item sucks", "gimme this and that, because I'm too lazy to get it my self" or "server crashed gimme some ph4t l3wt as compensation".
If ya don't like it PLEASE QUIT !!


---------------------------
/ignore ph4tl3wtd00d3
Tyndes - lvl 62 clr <Veeshan>
RE: HA HA
# Jun 13 2003 at 9:05 AM Rating: Good
your an idiot.

there is no reason what so ever these people shouldnt be angry at losing a possible days worth of play and work. if they wish to complain, id allow it. but when it comes to asking for compensation, that is a different story. i dont think you should get anything for it. stuff happens, hardware breaks down, and you have to fix stuff. its one of the risks you take in playing EQ, along with the risk that one day no one will play EQ no more and your character will become a vague memory and nothing less.
#Anonymous, Posted: Jun 13 2003 at 3:47 AM, Rating: Sub-Default, (Expand Post) you'd be singing a DIFFERENT tune if YOU played on prexus. bad stuff happens, people *****, its human nature. Sony SHOULD have backed up the server, they SHOULD have reimbursed people in some way (charm of prexus, anyone?) and/or they SHOULD refund people 10 bucks.
#Anonymous, Posted: Jun 13 2003 at 5:25 AM, Rating: Sub-Default, (Expand Post) It's just a game, but when I see the time I spent building up my char, I'm sure wouldn't like it to happen for me!
#Anonymous, Posted: Jun 13 2003 at 1:12 AM, Rating: Sub-Default, (Expand Post) blah blah blah blah blah blah and.......blah ohhh... and blah
Grow up
# Jun 13 2003 at 12:15 AM Rating: Decent
Sheesh it was a flipping crash. Come on now. ******** about reimbersment for a failure that was not there problem is like asking them for XP while your in trader mode, or logged off. Now if you think that sony created the problem, dont you think it would have heppened on all the servers...see there all exactly the same, except for stormhammer. Now im not gonna go in depth about any of the things that may have gone wrong. But what is sounds like is there was serious hardware failure. Now im done ranting and raving. I just had to insert my 2 cent

Minyumivia
Bard of the 53rd song.
Cazic Thule
RE: Grow up
# Jun 16 2003 at 10:03 PM Rating: Default
Well for one i put money into it...then there is this trust thing that if i am to give u money and pay for a service i should get what i payed for it. Another thing it is not just the monthly money it was buying the game in the first place. If u r to offer something to someone for a price u need to live up to that. They need to stop upgradeing so damn much that they can't handle the back up that might come... To be good is not always being updated but being damn good at what u do. Pally from vallon
#Anonymous, Posted: Jun 12 2003 at 11:36 PM, Rating: Sub-Default, (Expand Post) Okay, it's as simple as this, complaining just shows how big of a moron you are, they can't and won't do anything about it, live with it.
well...
# Jun 12 2003 at 11:34 PM Rating: Default
This sucks, certainly, and its obvious there's nothing they can do, so complaining is pretty pointless. But its frankly very surprising that their data isn't replicated in real time. Who, at that level, doesn't cluster their data servers? All my customer data is written simultaneously to two database servers, both with level 5 raids... and that doesn't use any additional system resources, though it does add to local traffic, which is easy to build for. The only way I would have to do a complete restore from backup would be if the building burned down, literally. Can't believe Sony isn't as fault tolerant as a little shop like mine...
RE: well...
# Jun 13 2003 at 8:51 AM Rating: Decent
Scholar
*
53 posts
Well, that would be ideal and im sure they have considered it, but as you say, that would introduce more traffic, something they have fought to reduce to improve the gaming experience. In a small network environment as yours (and mine here at work), the impact of clustering would be minimal, but on thier network where each "server" is 30+ servers, something like clustering or a fault tolerent solution would likely have a large impact on performance. Considering how often this happens (which is VERY rare), they probably made the right decision not to (yeah i know, this coming from a non Prexus player).

I do wonder if they have this on Legends, since they charge a premium and have certain "guarentees" on that server.

Edited, Fri Jun 13 09:29:37 2003
« Previous 1 2
Post Comment

Free account required to post

You must log in or create an account to post messages.