Last weekend, the v2.0 update to ArcheAge was deployed on Trion's ArcheAge servers. However, Glyph had other ideas and the weekend was fraught with issues. We caught up with Trion CEO Scott Hartsman to find out more about the issues, what will be done and plans for future updates.
As of this writing, Trion Worlds announced plans for compensation for those affected by the 2.0 launch issues. See the forum thread here.
MMORPG: First, let's start with the obvious: what happened with the ArcheAge 2.0 patch process, and what went wrong?
Scott Hartsman: Short version: Unexpected type of load on our account system that began just prior to the login rush, and had not been present through testing or prior game updates or game launches.
Long version:
The first half of the effort, the server transfer stage in the days prior, went off perfectly and all was looking optimal. The evolution launch itself didn’t, and required an 18 hour delay for EU and 24 hour delay for NA.
That unexpected delay in the Heroes Awaken launch was heartbreaking for the team as well. Everyone involved, me included, feels absolutely terrible that the delay happened at all and is extremely sorry for the inconvenience that people faced.
The amount of effort that goes into an effort of this type, in an open world game whose data is this uniquely complex, is something that many people aren’t familiar with – Just to share details: Six months of prep, a month of testing, 96 hours of concentrated final testing, both the XL games team from Korea and the Trion team all on site together in our Austin studio, the goal being to ensure that everything went off and all the issues of past launches were learned from. A planned rollout of 50+ steps on a checklist that all had to happen at the right time in the right order. That all went off just fine. From load, to fraud, to security, to data conversion and integrity, to updates of clients and servers and so on. All was looking perfectly solid.
What we can say is that once it finished, we had a consistent and extended period of externally driven account service usage, far greater than what we've seen in even larger full game launches or updates, which began just prior to the initial login rush and extended for quite some time afterward.
That caused blocks on game login, store use, account management, and anything else that touched the account system. We don't like using the word 'attack' without a full and thorough investigation, which is still in progress as we eliminate variables one at a time. The full and detailed post-mortem will take a couple weeks of investigation and research. That said, where the extra non-game load came from isn’t as important to us as knowing that it’s possible and that we need to be able to handle load of that pattern far better in the future, so people aren’t negatively affected when it happens again.
The second release was solid, fortunately. And we’re eternally grateful for everyone’s patience.
XL learned a lot from the prior releases and added the ability to roll back the servers in case of trouble. (The way their game services interact with our platform made that mechanically not workable in prior releases. In online games of all kinds, your character’s information and inventory is generally managed by the game. Things that your account is entitled to tend to be stored on a given game’s platform – This is how it works on systems ranging from MMOs to consoles. Same types of systems.)
Given the game upgrades since the initial release, we were extremely happy to able to reset it all back for maximum fairness. Fairness being the point of this whole release. We're happy that the plan for doing so nearly mirrored what the community was hoping for during the delay. It took a lot of effort from the developer to make sure the game could handle that. They did a great job here, and in working hand in hand with our database and operations teams in pulling it off, and it was incredibly well worth it.
We then focused on making sure we could launch successfully through this extreme load, while at the same time working with the game team ensuring that everything could be safely put back the way that it was. The development and production teams worked on that while the operations and platform teams handled mitigating the external load.
Aside from the account service load issue, we're happy to say that the rest worked as expected. As of the launch the game's in a great place, the tone in game is very positive, and there are more people playing right now than there were through all of 2015.
And in additional lessons from this: We now have an entirely new type of ‘external load’ to test against and ensure we can survive, both for future launches of other games and updates to existing games. Our platform and operations teams have been working on that for the last couple days, and have made great progress even in this short of time.
MMORPG: When did you know something was amiss, and decide to take the servers down for an additional 24 hours?
Scott: Ahead of the launch it was: “That’s a little odd, but it should be all right.” Things were reading a touch different than normal, but not explosive or into what we’d call bad. Nothing that appeared out of the normal range of a spike in one of the other games.
About 10 seconds in when the servers didn’t instantly spike to full it was, “This should be able to catch up. It has before. Let’s give it a few minutes.”
At 20 minutes when it didn’t catch up, and 10% of the expected population was on servers, the production team made the call to hold off and restart. If we couldn’t get people a fair crack, we weren’t going to do it. I completely agree with their call.
MMORPG: Will there be any kind of compensation for players that were affected by the extended downtime?
Scott: Yep. Specifics will be out on our forums today.
MMORPG: Why wasn't there a property limit put in place before the patch, so that the claiming issues that happened on Saturday couldn't happen?
Scott: Property claiming worked out according to the rules of the game, with the limit of 2 unbuilt houses doing its job. There was a property limit beyond that that was defined by the number of “full kit” houses that people received, which was equal to the amount of land they held on pre-evolution servers. Definite improvement over any past release.
Beyond that, GMs, the game’s developers, producers, and CMs were all on servers in person to witness firsthand and keep an eye out for any potential bad actors, which we’re happy to say didn’t occur. The main issue was access to the servers themselves.
We were very happy to see a community poll where the majority of voters indicated that they’d gotten back as much land as they previously had or more, and some who had never had land previously now do.
MMORPG: Given the rocky start of ArcheAge when it launched, why not stagger the server start times a bit, instead of bringing them up all at once?
Scott: This was absolutely too ambitious. We were trying to maintain NA and EU parity and not show favoritism one way or the other, as that tends to be extremely contentious. Our testing indicated it would be all right to do the global/simultaneous release. Staggering it in the final release definitely did help.
MMORPG: It seems that ArcheAge is one of those games whose patches just can't catch a break. Because of the nature of things like land claiming, it feels like there will always be these issues. What have you learned from launch and from 2.0 for future major udates?
Scott: This is really insightful. ArcheAge is indeed in a rare position here. One of the most compelling features of the game is also one of the most difficult ones to ship in a game at massive scale: Land ownership with objects and structures built onto it, many of which have growth states over time. With some of them being able to contain dozens or hundreds of other items. All in shared world space.
There’s a reason that many games don’t ship with a feature like this: It makes everything else about the game’s architecture, data, and so on, an order of magnitude more complex than not doing so. On the bright side, none of the issues that we had were related to that feature. Huge amounts of planning and proper execution saw to that. It was purely related to account system load.
MMORPG: Do you feel that your contract with XLGames and the limited control Trion Worlds has on development overall has negatively impacted the NA/EU user’s experience? It’s the first game you guys published, but didn’t develop, so it’s got to be double hard because the company is in a different country altogether. It’s a loaded question, I know, but what I mean is that the markets and playerbases are so divergent, it can’t be easy to walk this tightrope.
Scott: Genuinely, it’s gone both directions. They don’t do things the exact way we would, which is no secret. People can (and have) compared and contrasted similarities and differences between the way their dev team manages certain live aspects of games with how our teams do the same. From systems design, to monetization design, to update cycles, and so on.
In some ways differences are truly excellent: I guarantee that we wouldn’t have undertaken a seven year development effort to create a game with open world land ownership, even though a number of us would have loved to. That alone requires a different kind of thinking.
Other times, we have to argue for things that we feel are very important to our communities. We win some, we lose some, which is no different really than any working relationship between two companies. In the end we try to net out to more positives. As the teams work more closely together over longer periods of time, we end up mutually winning quite a lot.
Correct in that it’s never easy. But it is worthwhile. And it’s a lot of fun when it works.
MMORPG: Lastly, I'll leave an open ended one for you. What do you, and the rest of Trion Worlds want to tell your players about this update and the future of ArcheAge? The game's almost caught up to the KR version, so what's coming later in 2015?
Scott: This release absolutely brings the best of what ArcheAge has to offer, with more restriction-free building space available than ever before and more to do than ever before. With a lot of enhancements that came directly from user feedback – Overall improvements to systems like Sport Fishing and Salvaging, better ways to manage guilds with it also being easier to find others who have similar playstyles, and more.
The content teams have already been hard at work for weeks on the updates that will be coming in the future, both in AA 2.5 which includes new Arena modes and more, and some amazing seasonal events that have never been seen here. We’ll be happy to talk about all of those soon as we can.
In the meanwhile, we thank everyone for their patience in our needing the extra day, and we’re looking forward to getting some fun new comp packs out there too. Details to come soon on our forums.
