There was a tweet yesterday by VentureBeat's Jeff Grubb that pretty much solidified what I was feeling during New World's troubled launch yesterday. In a day filled with server queues and long wait times, it reminded the gaming internet (at least those who follow the long-time, well respected games journalist) that even large companies aren't immune to launch day woes.
If New World, the MMO from the company that owns most of the servers that host the internet, has login queues, I think we can probably dismiss the idea that this problem is avoidable for other devs.
— Jeff Grubb (@JeffGrubb) September 28, 2021
It's poignant mainly because when we think of Amazon, many of us think Amazon Web Services, the server infrastructure that basically powers much of the internet we know and have a love/hate relationship today. MMOs one day one are never pretty. There are long wait times, server instability, queues and more seemingly with every major online game nowadays as well. Even Blizzard, a company worth billions and has decades of online game operational experience, has struggled with its latest release of Diablo II: Resurrected.
However, I had some hope New World would be different, mostly due to the company and their reputation for scalable server services backing it up. However, if you follow me on Twitter (and if you don't, no worries - you're not missing too much), you'll notice it was a point of joking and consternation all morning.
Opening the floodgates
New World was bound to be popular. If anything, New World's mad rush of over 700K concurrent players during early morning West Coast time here in the US was proof that MMO fans are eager for something new and interesting - even in a year that has seen multiple new MMO releases. The stregnth (and financial backing) of Amazon Games' marketing coupled with the fact that many of the world's top streamers have effectively been added as a wing of that marketing arm built up excitement. Sites, like ours, wrote about New World every chance we could - mainly because it was news, but there are always the traffic and SEO implications of articles we have to write (we do literally operate on traffic - that's the game).
So when New World threw open the floodgates on September 28th, the fact that there were a ton of people trying to access the MMO and start their journey was to be expected. Amazon expected it - they had pre-order numbers behind them as well as survey data from the previous alphas and, most recently its open beta test. However, the servers themselves seemed to buckle under the weight of all those people trying to dash towards their predetermined region and world - some trying to link up with their favorite streamer, others (like us here at MMORPG) trying to set up shop far away from what we hoped would be the worst of the queues on launch day.
At launch, Amazon hit the ground running with over 170 servers, though the amount of players those servers could support was found lacking. With a population cap of 2000 on each, those servers filled up fast as the concurrent player count skyrocketed to over 600K players initially, hitting over 700K at its peak (if you're keeping score, those 170ish servers could support about half the 600K players trying to gain entry).
Server queues hit into the thousands - some in the tens of thousands, with players getting disconnected from queues throughout the day. Amazon, after launching on the West Coast of the US, thanked fans for the response and vowed to spin up more servers and up capacity - something the team did throughout the day, adding 28 servers to the East Coast North American region at one point in a single pop. To their credit, those new servers as well had fast queues, getting players into New World quickly. This morning was no different, and while my original server was still sitting with legnthy queues, the world my friends and I moved to connecting was a breeze.
However, the damage was done. It was another MMO that was seemingly unprepared in many players' minds for the demand that followed it during its multi-year long marketing and development cycle. Players took to Steam to review bomb the MMO for server queues while I spent most of the morning working and both joking - and complaining - on Twitter about the launch day woes.
Once I was finally in our chosen server, I was afraid I was going to get disconnected or be timed out for going AFK - I did have our normal duties here at the site to worry about, as well as being a single dad, at some point I'd have to, you know, parent. The server was full of like-minded players, with scores of them setting their characters to auto-run into walls in an attempt to trick the AFK timer and stay logged in throughout the day.
In the evening more servers were added and my group and I were able to find a lower populated server to regroup on. Amazon, to their credit, improved queues on these new servers, with friends and other players able to navigate them in a matter of minutes instead of the hour long soujourns our GPUs took rendering a login screen throughout the day.
Even large companies can stumble
Amazon isn't new to the server infrastructure game, which is why it was so surprising the server issues felt so exacerbated here to me. New World was always going to have launch issues - no MMO or seemingly any game with a major online component launches without them now - but the extent took many, myself included, by surprise. I expected that a company that sales scaleable server technology as one of their core services - as well as provides hosting and servers for literally a huge chunk of the internet - could mitigate this as much as possible. Yet as I rounded the corner of my third hour in a queue I started to plan out whether or not I'd be able to start my review yesterday at all.
Simply adding more servers won't fix the issue, as Phoenix Labs senior programmer Chris Pollock told IGN in an interview in August. It's more complex than that.
"Every live game falls over day one[.] The internet screams, 'Add more servers,' and the number of servers is quite literally never the problem.
"People think of it like: a server can handle X players, Y people want to play, so Y / X = number of servers = problem solved, right? Simply: no. There are at least four more layers of complexity to a live game, and each one amplifies any of the previous layers' problems."
Simply spinning up capacity, or just launching with even more servers, doesn't solve the fundamental issue that could plague these servers: demand. Even if developers are expecting millions of players to descend on a game, that doesn't mean the bandwidth is there to handle that many players at one time. Additionally, having tons and tons of servers can add to cost - though one could argue Amazon can afford it here - especially as many of those servers drop drastically in population weeks after launch when the hype wears off.
What got me specifically, though, during the launch on Tuesday was how a myriad of choices seemed to affect the end result: hours long queues and mixed reviews on Steam. Amazon's New World has been one of the most hotly anticipated titles in years, seemingly. With all of the data available one could argue more servers should have been available at launch to cover projected players - or at least spin up capacity on the servers themselves as Amazon has done today. 2K players on a server seems low for a game billing itself as an MMORPG, but with so many servers launching with New World, the thought might be that 2,000 players on each world provided balance, especially in a PvP-focused game.
However, the decision to tie character names to worlds globally versus tying them to the specific server the character was made on compounded issues from the get go. In an effort to reserve long-held player names - identities that have traveled with MMO players from game to game, sometimes for decades - many US-based players hammered the EU servers when they launched in order to reserve their name globally to use when the North American servers spun up the next morning.
I have to wonder if this decision, with the lower than you'd expect player population cap on each world - and then just overwhelming demand during early hours - not even peak ones - just created a situation that wasn't entirely anticipated. Amazon isn't a blind or dumb company, and the development team at AGS is a talented bunch of industry veterans, many of which have gone through MMO launches as a developer before. However, as I sat wondering why the launch issues felt exacerbated on launch day, a large part of this is due to the fact that I just simply expected a better roll out from the company that powers much of the internet's web services already.
I don't really know what the answer would have been, and people infinitely smarter than I am in this realm seemingly didn't either. Is it more servers, higher server pop, or simply adding more lanes for people to connect on, mitigating bottlenecks? I don't know. But for a parent company that specializes in server solutions to have one of their product that suffered such widespread server issues, it felt more exacerbated than typical launch day woes.
New World's launch issues aren't unique, nor will it be the last MMO to deal with them. But they do feel like the first time a company had the resources and built-in institutional experience to potentially mitigate them as much as possible. The real test now is to see how AGS responds throughout the week and into the weekend, and whether or not we'll see a return of long queue times, or if those are already a thing Aeternum adventurers are leaving in the Old World.
