Everything you said depends rather heavily on the servers themselves essentially growing in size to sovle their problems. Even more so it's largely problems that can only be solved if the failings are piecemeal. The loss of part of a server cluster might be recoverable, but that doesn't outright solve crashed servers nor does it solve a power outage.

Solves some problems, doesn't solve this problem. You aren't running any server if you aren't powering it or if the hardware as a whole goes down.

EDIT: Also making a server making a mirrored backup doesn't solve the problems associated with server load or how the data is backed up to the servers. The information has to make it to being bakced up in the first place. I even referred to server backups before, though I did not call them mirrors.

EDIT2: Apparently you actually acknowledged this fact previously, stating this following your diatribe on the probability of something to happen with failsafes mentioned.

"Short of site damage to fire or flood or some such disaster."

So...guess what?

Or does the power getting cut not count?

Originally posted by fyerwall

Actually it IS something that could have been prevented if they had the proper infrastructure in place. This just smacks of "How cheap can we go?"  And this is also something that can be fixed, given that the company cares and is willing to make the upgrades.

This is wild and unfounded speculation. No complex system is perfect and all have a risk of failing. If you invest resoures wisely you can increase the availability and robustness of a system, but costs increase signficantly as you approach 100% while the returns diminishes quickly. This is also a game. While a rollback sucks, it's not a big deal in the overall scheme of things. This isn't an air traffic control system or a bank. It's certainly a better outcome compared to loosing your character completely.

Generally the way the character save feature works, and I don't know how GW2 is implemented, is the player data is cached for performance reasons because writting to a database is expensive. The player data gets check pointed at regular intervals or specific events such as leveling or quitting and then it is written to a persistant database. In the olden days you could tell when a check point happened because the server would pause for moment. I've always seen the player data cached in the server application itself so if the server application crashes all data is lost up to the last checkpoint and you have the dreaded rollback. I wouldn't expect the game server application infrastructure to be highly available, the critical databases - yes, most of the network - yes, but not the servers running the game itself. There's just too much rapidly changing and performance sensitive state on the game server for a highly available solution to be practical.

Originally posted by Nadia

 

https://forum-en.guildwars2.com/forum/support/account/Tickets-for-Review-5-days-and-older-merged/page/28
GaileGray

    Galadeon.6513:

        GaileGray.9587:

        No we are unable to restore gold. It’s simply not something that we have the tools to do, and the account restoration system that would roll-back the entire account is not yet available. I’m very sorry for the loss, and I know we all look forward to being able to help with these issues in the future.

    Sad, very sad that your programmers would not put this BASIC MMO functionality into the game. There should be a plan in place to help people that get hacked.
    Note to game makers – make sure that your future CS has the tools it needs to help people.
    #NotAAAservice

You may not be aware, but spawning gold and handing it out like cupcakes — even if “basic MMO functionality” — isn’t good for a game’s economy. If you lose a dollar, the government doesn’t print up and hand you another one — that would cause huge inflation.

To address the issue of hacked account, we soon will be offering account restorations that will “roll back” an account to its pre-compromise state. The issue will be verified, and the hacker will be tracked and and all account involved in the hack will be closed. Then, when the hacked account is rolled back, the gold is not duped and the economy isn’t damaged.

Oh my what a mess up.

Originally posted by bansan
Last night around 10 PM PST, the servers crashed for about 5 minutes, and I think pretty much everyone on at that time experienced a short rollback. Not fun, but it happens. You really have to laugh at people who think this doesn't happen in other games, or it is because of worse customer service. This happened to me in WOW, LOTRO, and some other crappy f2p games. None of those game offer any, er, roll forward for items or xp. Some did offer 2 times xp for the entire server.

Playing mmos for 7 years and I have never heard of accidental roll bavk with no restor. Even BioWare restored everyone when they rolled back patch 1.2.

---

Originally posted by Deivos
Everything you said depends rather heavily on the servers themselves essentially growing in size to sovle their problems. Even more so it's largely problems that can only be solved if the failings are piecemeal. The loss of part of a server cluster might be recoverable, but that doesn't outright solve crashed servers nor does it solve a power outage.

 

Solves some problems, doesn't solve this problem. You aren't running any server if you aren't powering it or if the hardware as a whole goes down.

 

EDIT: Also making a server making a mirrored backup doesn't solve the problems associated with server load or how the data is backed up to the servers. The information has to make it to being bakced up in the first place. I even referred to server backups before, though I did not call them mirrors.

 

EDIT2: Apparently you actually acknowledged this fact previously, stating this following your diatribe on the probability of something to happen with failsafes mentioned.

"Short of site damage to fire or flood or some such disaster."

So...guess what?

Or does the power getting cut not count?

---

I am not following this. I'm sorry. Either you don't fully understand how this technology all ties in or I'm not understanding what you are trying to say. (I'll accept the 2nd)

Getting back to the initial topic, there is no excuse for Arenanet to say they have no ability to do rollbacks if they are using the proper technology.

As for power outages, there are fail-safes for that too.

Originally posted by GeezerGamer

They absolutely do. You put a server operating systems on a VM and keep the database files in a SAN with Raid 10, mirroring and parity, You split up your database files in a SAN with each portion on it's own dedicated spindles, You keep the SQL operating system in a VM or at least on their own drives, you keep the Databases on a 2nd spindle and the transaction log files on a third. In the event of a catastrophic failure, you can spin up a VM in minutes from a copy on file. While you have the mirrored parity system automatically restoring the lost database files. The mirror immediately cuts over while the parity rebuilds the missing data. and there is little to no interruption in service. Aside from some customers complaining about some lag anyway.

I mostly agree except for one important detail - GW2 is actually using a much more complex internal worldsystem architecture infrastructure than any MMO currently on the market. Dynamic Overflow Zones means the "traditional" concept of a Server doesn't exist in the game.

There's no question they need to improve their capabilities with regards to failover and data recovery, but to just "assume" that the architecture or inter-system/server communication is the same as RIFT is simplistic at best and misleading at worst.

Data loss sucks and obviously it's nothing that ANet wants to happen. But to assume that the issues they encountered were simple or easy to fix just isn't fair. SWTOR had the same and even worse problems in what seemed to be a much simpler architectural design.

You also fail to account for player/overall game impact in your analysis. How many people were impacted by this? How many weren't but would be impacted by a full recovery from logs? What kind of strain would be put on the systems by instituting a "global update" or even a query attempting to select just the impacted characters?

Even in the architecture you described up above if there is a communication breakdown between the Transactional Server and the Logging System you can't recover. And even if you have the logs, it is a non-trivial and intensive process to update only those accounts that were impacted.

Once again, I want ANet's Systems to improve and I certainly feel for the people who suffered data loss. Just don't pass this off as "simple" because it isn't.

Originally posted by GeezerGamer

 

---

Originally posted by Deivos
Everything you said depends rather heavily on the servers themselves essentially growing in size to sovle their problems. Even more so it's largely problems that can only be solved if the failings are piecemeal. The loss of part of a server cluster might be recoverable, but that doesn't outright solve crashed servers nor does it solve a power outage.

 

 

Solves some problems, doesn't solve this problem. You aren't running any server if you aren't powering it or if the hardware as a whole goes down.

 

EDIT: Also making a server making a mirrored backup doesn't solve the problems associated with server load or how the data is backed up to the servers. The information has to make it to being bakced up in the first place. I even referred to server backups before, though I did not call them mirrors.

 

EDIT2: Apparently you actually acknowledged this fact previously, stating this following your diatribe on the probability of something to happen with failsafes mentioned.

"Short of site damage to fire or flood or some such disaster."

So...guess what?

Or does the power getting cut not count?

---

 

I am not following this. I'm sorry. Either you don't fully understand how this technology all ties in or I'm not understanding what you are trying to say. (I'll accept the 2nd)

Getting back to the initial topic, there is no excuse for Arenanet to say they have no ability to do rollbacks if they are using the proper technology.

As for power outages, there are fail-safes for that too.

No failsafe is 100%. I have personally seen an entire data center go down. Both their main generators and back up generators failed.

---

Originally posted by Stx11

---

Originally posted by GeezerGamer They absolutely do. You put a server operating systems on a VM and keep the database files in a SAN with Raid 10, mirroring and parity, You split up your database files in a SAN with each portion on it's own dedicated spindles, You keep the SQL operating system in a VM or at least on their own drives, you keep the Databases on a 2nd spindle and the transaction log files on a third. In the event of a catastrophic failure, you can spin up a VM in minutes from a copy on file. While you have the mirrored parity system automatically restoring the lost database files. The mirror immediately cuts over while the parity rebuilds the missing data. and there is little to no interruption in service. Aside from some customers complaining about some lag anyway.

---

I mostly agree except for one important detail - GW2 is actually using a much more complex internal worldsystem architecture infrastructure than any MMO currently on the market. Dynamic Overflow Zones means the "traditional" concept of a Server doesn't exist in the game.

There's no question they need to improve their capabilities with regards to failover and data recovery, but to just "assume" that the architecture or inter-system/server communication is the same as RIFT is simplistic at best and misleading at worst.

Data loss sucks and obviously it's nothing that ANet wants to happen. But to assume that the issues they encountered were simple or easy to fix just isn't fair. SWTOR had the same and even worse problems in what seemed to be a much simpler architectural design.

You also fail to account for player/overall game impact in your analysis. How many people were impacted by this? How many weren't but would be impacted by a full recovery from logs? What kind of strain would be put on the systems by instituting a "global update" or even a query attempting to select just the impacted characters?

Even in the architecture you described up above if there is a communication breakdown between the Transactional Server and the Logging System you can't recover. And even if you have the logs, it is a non-trivial and intensive process to update only those accounts that were impacted.

Once again, I want ANet's Systems to improve and I certainly feel for the people who suffered data loss. Just don't pass this off as "simple" because it isn't.

---

GW2 code being the most advanced (Which from what I've seen, I have my doubts) has nothing to do with the quality of their infrastructure.

I never assumed their infrastructure would or should be the same as Rift. I am saying, the technology is there. Whether they use it or not is a different story. I was simply providing an example of how such technology works. A hypothetical situation. And yes, I tried to keep it simple.

I'll give you your point on the fix might cause bigger issues than what it corrects. That's possible. But again, It's case by case. Anet isn't doing this.

You are also right about data loss, nothing's perfect and DR setups should be within reason and not go all out. But there comes a point where basic functionality is being disregarded across the board and for the most part your examples of the exception would cover individual situations on a case by case basis and would hardly justify Anet's blanket statement of no restorations.

I am also not forgetting something else, None of this may even be infrastructure. These could very well be because they simply don't have the staff.

Pretty much Xzen said. I understand the architecture fine, my point is that even with all that it doesn not solve the issues that they experienced. They noted on twitter that the system went down due to an outage. It wasn't one database either, login servers were affected too.

And yes, there are things that can be done to absolve power outages. I even mentioned as much before, much like other things now being said. I however also note that's not an absolute defense.

Anything else is demanding they invent a perfect system that just doesn't exist.

All in all it falls back to my comment made previously, unless they have the data or at least a snapshot of what happened at the point of the crash, having the tools to recover information is moot because falsities can be made in addition to imbalances of the game state unless they can restore everything.

Doing rollbacks I'm pretty certain was never the problem, it's recovery from an unintended rollback in this instance.

As for why they don't do individual rollbacks yet, I have to assume that's more to do with tracking all the assets. The data and tools are there, they just aren't organized for the devs to read it out and handle easily. Unless they get to read large chunks of data from multiple clients to parse the information they need for rollbacks, bans, etc.

Originally posted by GeezerGamer

GW2 code being the most advanced (Which from what I've seen, I have my doubts) has nothing to do with the quality of their infrastructure.

I never assumed their infrastructure would or should be the same as Rift. I am saying, the technology is there. Whether they use it or not is a different story. I was simply providing an example of how such technology works. A hypothetical situation. And yes, I tried to keep it simple.

I'll give you your point on the fix might cause bigger issues than what it corrects. That's possible. But again, It's case by case. Anet isn't doing this.

You are also right about data loss, nothing's perfect and DR setups should be within reason and not go all out. But there comes a point where basic functionality is being disregarded across the board and for the most part your examples of the exception would cover individual situations on a case by case basis and would hardly justify Anet's blanket statement of no restorations.

I am also not forgetting something else, None of this may even be infrastructure. These could very well be because they simply don't have the staff.

Just to be clear I didn't say "advanced" I said "complex" - with the way Guesting and Overflow Zones were designed to work (and even just the Character Naming Policy) Character Data is handled "globally" in their system. That has implications both at a software and hardware level. RIFT, WoW, pretty much any MMO I can think of other than EVE can (and I'm sure do) segment their physical Character Databases by Server. GW2 can't. There are both strengths and weaknesses to the design. The potential impact of a single point of failure is much higher in GW2's design which is a weakness.

Regardless, I think your second point is likely closer to the truth for why they have made a blanket statement about character/item restoration.

---

Originally posted by Stx11

---

Originally posted by GeezerGamer GW2 code being the most advanced (Which from what I've seen, I have my doubts) has nothing to do with the quality of their infrastructure. I never assumed their infrastructure would or should be the same as Rift. I am saying, the technology is there. Whether they use it or not is a different story. I was simply providing an example of how such technology works. A hypothetical situation. And yes, I tried to keep it simple. I'll give you your point on the fix might cause bigger issues than what it corrects. That's possible. But again, It's case by case. Anet isn't doing this. You are also right about data loss, nothing's perfect and DR setups should be within reason and not go all out. But there comes a point where basic functionality is being disregarded across the board and for the most part your examples of the exception would cover individual situations on a case by case basis and would hardly justify Anet's blanket statement of no restorations. I am also not forgetting something else, None of this may even be infrastructure. These could very well be because they simply don't have the staff.

---

Just to be clear I didn't say "advanced" I said "complex" - with the way Guesting and Overflow Zones were designed to work (and even just the Character Naming Policy) Character Data is handled "globally" in their system. That has implications both at a software and hardware level. RIFT, WoW, pretty much any MMO I can think of other than EVE can (and I'm sure do) segment their physical Character Databases by Server. GW2 can't. There are both strengths and weaknesses to the design. The potential impact of a single point of failure is much higher in GW2's design which is a weakness.

Regardless, I think your second point is likely closer to the truth for why they have made a blanket statement about character/item restoration.

---

I apologize, I should have paid better attention to what you said.
I won't disagree.

Originally posted by bbbmmmlll

Originally posted by fyerwall

Actually it IS something that could have been prevented if they had the proper infrastructure in place. This just smacks of "How cheap can we go?"  And this is also something that can be fixed, given that the company cares and is willing to make the upgrades.

This is wild and unfounded speculation. No complex system is perfect and all have a risk of failing. If you invest resoures wisely you can increase the availability and robustness of a system, but costs increase signficantly as you approach 100% while the returns diminishes quickly. This is also a game. While a rollback sucks, it's not a big deal in the overall scheme of things. This isn't an air traffic control system or a bank. It's certainly a better outcome compared to loosing your character completely.

Generally the way the character save feature works, and I don't know how GW2 is implemented, is the player data is cached for performance reasons because writting to a database is expensive. The player data gets check pointed at regular intervals or specific events such as leveling or quitting and then it is written to a persistant database. In the olden days you could tell when a check point happened because the server would pause for moment. I've always seen the player data cached in the server application itself so if the server application crashes all data is lost up to the last checkpoint and you have the dreaded rollback. I wouldn't expect the game server application infrastructure to be highly available, the critical databases - yes, most of the network - yes, but not the servers running the game itself. There's just too much rapidly changing and performance sensitive state on the game server for a highly available solution to be practical.

Its not wild and unfounded. I work with this 365 days a year.

Originally posted by MosesZD

Originally posted by Nadia

Originally posted by neorandom

Originally posted by Nadia

        GaileGray.9587:
the account restoration system that would roll-back the entire account is not yet available. I’m very sorry for the loss, and I know we all look forward to being able to help with these issues in the future.

stop going on suspect webpages and downloading keyloggers because they promise to give god mode in pvp?

what I posted was ANETs explanation that they have zero support for character restores or rollbacks

 

if you read the Op, you would know that he lost levels - and was not hacked

 

how does a hacker make you lose levels ?

 

I don't even believe the story.    This is some weird, isolated charge of malfeasense and incompetecy that is, on the face of it, unbelievable.

After reading your post history i am not surprised that you don't believe my 'story'. Do i care? nope.

ANET has already tweeted about this and i wasn't alone to suffer from this, several topics have been locked on official forums and some are still opened. Your own denail is not my problem.  Anet has made it very clear they can not undo roll back so what is lost is lost.

Originally posted by Amjoco

Originally posted by Kyleran
B2P, no monthly sub. You can't expect the same level of customer support as in other titles. Remember in the end all you lost was a little bit of time and some pixels. GW2 is all about having fun and not acquiring items or progression, right?

Over $1300 for WoW subscription the last several years is not worth the support I got out of them. I know one time I spent over an hour in queue on the phone trying to get my account back, and that was just one episode.

You are right, it is about having fun and I think once the dust settles the support will be fine at ArenaNet. Currently they may be a bit busy! :)

 

Did you get your account back though?  I have not had an account hacked but I have had friends that have and yeah its a pain in the ass to sit on the phone for a hour but they all got their stolen stuff back.  Seems like a hell of a lot better CS then not hearing directly back from the company and reading sorry your screwed on twitter.  Lets face it ANET does not know what they are doing when it comes to AAA MMOs it is out of their league.  Honestly I think this is what has given Blizzard lasting power more then anything else.  They started small and were able to grow as the game grew into a company able to hand multiple millions of players.  These other companies are being thrown into the deep end and it has shown in everything from game design to balancing to behind the scenes code that they left out like this not able to restore chars and items to CS to pretty much everything.

The more games come out the more I think for a huge player to come along it will not be a huge release game but a game that starts small and grows over 3 to 4 years time.  Or a huge hyped game that fails out of the box and rebounds over a few years.  This notion that some great game is just going to be ready and launch perfect and kill everything in its path is becoming more and more unlikely to me each day.

First time I hear anything like this. Don't know how this came over you. Even less ideas on how to proceed since Anet has already said they don't support account rollback, let alone rollforward if you will. I can only sympathize and hope you weren't running some 3rd party app that could have cause the servers to block saving your progress.

Only you know what was running at the time. You seem to be all alone in this matter.

Mind posting the server this happened on and the server times if they're different from yours?

Their account support forum has thousands of posts about hackers, rollbacks, etc.. thousands and thousands of accounts hacked.

Their response to all of them.. "we don't have tools to restore anything."

For an MMO this is unacceptable.  I can't honestly defend any game, no matter how good, if they can't have basic customer support methods like character restoration for hacked accounts.

I like GW2, but what they did with my friend's account was amateur at best.

First, they allow you to change e-mail/password without confirmation (they actually send you an e-mail saying "This was changed, we hope it was you!").  They have no phone or e-mail support.  You have to tie your game account to your support account.

It took 2 days and 4 updates from them to get the account back in her hands.  In this time they asked the same questions 3 times (even though they were answered each time).  They wanted a CD Key which wasn't possible (because it was digital and the e-mail for the purchase was lost in a PC crash).  It was a massive undertaking and excercise in frustration to get them to do anything.

I hope they can fix it going forward.. there are some basic things they need to do like make let you dispute e-mail/password changes, allow you to re-send the IP security confirmation (many people never got the e-mail and there's no way to get it resent).  Simple things that should be priority.

How do you lose an e-mail from a client crash?

...PEBKAC?

Are you certain it wasn't so hard for your 'friend' due to user error in any part?

I know the system itself is indeed missing vital components at the moment. They did say they ar emplementing rollbacks. You can actually dispute problems with accounts(assuming you keep the applicable data floating around, the recipt/key e-mail shouldn't be the only means of confirmation you have and you likely have multiple points of confirmation depending on how you paid for the game).

Aside from that condolances I guess to a terrible rash of luck for your friend and I hope they have better luck in the future.

Rollbacks in AAA title.

Woah.