Sanya Weathers's MMO Underbelly: Server Stability Blues
Each Friday, the former Director of Community Relations for Mythic Entertainment pens this column that focuses on some under-served MMO story. In her debut, she tells us the Server Stability Blues.
A short time ago, I was playing my favorite MMO, and there were… issues. My server is somewhat overcrowded compared to some of the other servers in the list, and it shows. We periodically froze in place. In the north part of the zone, melee simply didn’t land and the monsters got the icon for “hey, I can’t hit you.” At least, until you got out of range and got hit with the critter’s ranged attacks, which meant that anyone of the melee classes simply couldn’t play. All was fine in the southern end of the zone, until the entire server wobbled and dumped a third of the players.
OOC was, predictably and understandably, filled with frustration.
Then came one particular outburst. “(Insert game developer) doesn’t *care* about server stability!”
Oh, god. Someone found the stupid dial on my server and jacked it to eleven.
Server stability is sometimes the only thing a game developer cares about. The difference between a big game company and a small game company is that the tech department head at the big company isn’t getting his rear end out of bed at 2 AM when things fall down go boom. He snorts in his sleep, having felt the disturbance in the force, and falls back into slumber knowing the newbie is the one blearily making his way to the colo.
At a small company, the department head is the whole department.
Either way, the entire job of some poor schmo is to get up and get the servers back up, regardless of weather, time, or degree of success being had with the only girl from miniatures gaming night at the hobby shop.
So why might there be persistent issues with server stability, despite it being the top and possibly only priority of the game studio?
- The server is at the top of the list of options, alphabetically. No matter how many gentle suggestions appear in the announcement box for new players to please for the love of god choose Zifferelle as their new home, new players wind up picking Abericious. In this scenario, Abericious and Berisia are the only unstable servers. There is no way to fix it besides mandatory server transfers.
- The server has a recognizable name. LOTRO’s first two servers are “Arkenstone” and “Brandywine.” Guess which one is overloaded. Hint: It’s the one recognized by the most casual of Tolkein fans.
- The last patch put in a highly desirable feature, one that measurably increased sales. A feature cited by market research groups as THE reason to resubscribe, or extend a subscription, or recommend the game to a friend. In testing, all was well. In release, not so much. Either the test server didn’t simulate the load well enough, or the problems didn’t build up enough to be noticeable for ninety days – sixty days longer than the new feature sat on the test server. Whatever the problem, the studio is now between a rock and a hard place. Remove the feature and lose money, or fail to fix the problem and lose money. Most people choose to work frantically on fixing the problem, but are hesitant to promise when the issue would be fixed… because if it were an easy fix, the players would never have noticed the problem at all.
- The game might be more than a few years old, on its tenth group of server programmers. The resulting layers of code would take a genius with infinite free time to puzzle out. If the game is old enough, the server programmers are the youngest, greenest, newest hires not considered experienced enough to work on the unannounced Future of Gaming product. So the person least able to deal with the legacy is the one who has to try. This would seem to contradict my assertion that server stability is all the company cares about, but it truly doesn’t. In this scenario, the server programmer is one of the only people left on the team at all.
MMO development is a lot like a duck swimming. Players see the calm, peaceful duck skimming along the water, sucking up giant amounts of subscription money kelp. We don’t see the frantically paddling feet trying to keep the whole thing afloat. So, when your server wobbles, be angry. Be frustrated. And raise hell if there’s no acknowledgement of the problem on the official communication channels within an hour. But don’t, DON’T prove your ignorance of the industry by claiming that the company doesn’t care.
Next week, I’ve got some discussion with an actual server professional to share with you. Tune in then!
More Developer Perspectives Features:
More Columns:
More Features:
Good read, I would actually like to see a bit more of a behind the scenes view of MMO's, thanks!
0/\0
Excellent read! I always wondered how people can think that devs don't care about things that will lose subscriptions for their game...
There was a time that I didn't think very highly of Ms. Sanya (I know, like she cares, right?
). But, after being introduced to Tasos, she ain't so bad. I mistook feisty for something else, I suppose. Anyway, good read. I don't necessarily agree with her points, but that's the great thing about living in a free society. 
I WANT MORE OF THIS!!!
These types of stories, the behind the scenes look and the feelings/thoughts of the people who keep the game going, up, and patched would be GREAT!!
I want server admins, update coders, and even the freaking janitor who has to keep the servers clean.
Thanks for a great article.
These are the kinds of articles I like.
Great choice in getting Sanya on here.
Tweety! Awesome...she will be very welcomed addition here and her first little article is a winner.
I also enjoyed this article.
Two more reasons why people choose a more highly subscribed server:
-- It is more likely that a new player has family, friends, enemies, etc. already on a busy server than on an empty one.
-- Some people have had bad experiences with games with low server populations, and so they deliberately choose servers known to have higher activity (livelier).
Until developers have figured out how to deal with these problems, admonitions to use a low population server will not be as effective.
I've been following Sanya ever since I joined DAoC at launch.. Wasn't around for her EQ years, but still a big fan. For those who liked it, might I recommend her other sites/blogs/fun factories:
http://www.examiner.com/x-5353-MMORPG-Examiner
http://eatingbees.brokentoys.org
I knew I forgot something! Updated her bio to include the linkage :)
I'd say right now, after the server order listing reason this would be number two on the list.
In the past, I used to chose servers with medium loads. But experience with several games has shown that more is better, and even if it means lag/queues in the early months, if I plan on playing a game a long time, I'll normally never go wrong chosing one of the most crowded servers.
Thats why it really "Grinds my Gears" [TM] when developers try to thwart me by either not listing the server population or trying to trick me by listing them all as "mediium". (looking at you, AOC)
This was a good article and something I took for granted. I never thought I played a game where the Dev's didn't care about stability, unless it was some F2P game or something.
But what does disturb me is dishonesty. No point to design a game with giant 200 vs 200 battles knowing a large majority of your player base will lag out/DC and apparently, theres no real fix for it based on current tech.
CCP/EVE have been working on the issue for 5 years, and made great strides and yet, many will tell you, there still are lag issues to be mastered. At least CCP doesn't hide from it or deny it exists, and they try very hard to fix it. I wish more companies would be honest in this way.
Anyways, grind off, I'm glad to see Sanya added to this site and I look forward to more articles.
Agreed. A great article and useful info. Nice work, Sanya!
I think you over looked the number one reason why some games struggle with server stability. That being the game was never completely stable to start with.
Using Lotro as an example- It never had a stress test that it didn't fail. Lotro's a great little game, solid content, solid community, great graphics and you get more than 50 people close together and the game chunks and lags. Get over a 150 people close together and the zone wobbles, add other server stresses (peeps signing in, loading zones etc) Lotro crash and burns. If lotro had a meeting area similiar to other games like WoW's city Ironforge, Lotro's server would never boot up. Lotro's decision to go high graphic sunk server stability from day one. Lotro hand cuffed itself. Instead of revisiting their graphics they make sure their game design keeps people spread all over the map. It worked wonders for them for all this time. Now with an aging game and most players all at endgame content the player base is jammed into smaller and smaller areas (eg 21rst Hall sorta mini Ironforge) and the game stability has diminished. The current free trials just add that extra stress that pushes their servers over the red line.
I used to be that poor bugger that got the 2 a.m. call go reboot such and such. It sucked! But what really sucked was that most times I had to run downtown to reboot such and such was that it would have been prevented if someone was inhouse monitoring instead of a script to ping everyone's pager that the servers crashed. You want to run your company 24/7 then have the support sitting there 24/7 scaled to server load.
The part on the programmers I agree with. But still, that's not the players problem. That still is the games company's responsibility to retain talent and document absolutely everything. As well, companies still think its the cheapest route to outsource and it can be. Until the duck feathers hit the fan and nothing is documented and no one on staff really know what depth the outsourced code reached.
Players are payers. Scream bloody murder on any and every issue you might have. Because if you don't you'll never get a 'ticket number' and if you never have a 'ticket number' it will rarely if ever be looked into. Gaming companies want to spend money forward not spend money backwards to fix stuff. You pay to play, whatever game you spent your money on keep that company on tenderhooks until your issue is resolved or take your hard earned cash elsewhere.
As allways nice to read something by Sanya. Being that it is Fryday think I will get a slushy tonight and toast to fond memorys of fighting for the realm.
/trollon
Wow, Kyleran, for an elite member of this site I would have expected better of you than: '...unless it was some F2P or something.' I hereby Knight thee Troll of the Realm. Your post is hereby summarily dismissed!
/trolloff
At least the rest of the responses, so far, have been enlightening, especially the ex server-maintenance chap.
Well written article, and ideas, I agree with most of what she says, but I have to admit, I love to play Devil's Advocate:
I'd have to say the most important thing anyone in any company should be worried about is profit. Everything else comes in second to that, however, server stability for me is definitely an EXTREMELY close 2nd.
"The server is at the top of the list of options, alphabetically. No matter how many gentle suggestions appear in the announcement box for new players to please for the love of god choose Zifferelle as their new home, new players wind up picking Abericious. In this scenario, Abericious and Berisia are the only unstable servers. There is no way to fix it besides mandatory server transfers."
I'd have to say a real quick fix for this is to leave the list of servers unsorted and provide a search feature to find your server or a separate box with a list of servers for which you have an avatar. Then just rotate the unsorted list every time a new player joins, or for a true bell curve effect randomize the list every time. This way a player can always have access to a specific server if they want one, but newer players that feel the need to just start right away will be able to get into the server at the top of the list which, hopefully will balance the load more. The only restriction is that you wouldn't want players to sort the list, but that's not the issue that's presented in that bullet point.
The server has a recognizable name. LOTRO’s first two servers are “Arkenstone” and “Brandywine.” Guess which one is overloaded. Hint: It’s the one recognized by the most casual of Tolkein fans.
This one is just the choice of the LoTRO developers, they could have just as easily named all the servers with obscure or non-obscure names or simply Server1,Server2,...ServerN. Granted this isn't "cool" or "hip" when it comes to the MMO community, but there's fixes for that as well.
The game might be more than a few years old, on its tenth group of server programmers. The resulting layers of code would take a genius with infinite free time to puzzle out. If the game is old enough, the server programmers are the youngest, greenest, newest hires not considered experienced enough to work on the unannounced Future of Gaming product. So the person least able to deal with the legacy is the one who has to try. This would seem to contradict my assertion that server stability is all the company cares about, but it truly doesn’t. In this scenario, the server programmer is one of the only people left on the team at all.
Good point here, but it all comes down to game design when it comes to extensibility and scaleability. If you plan ahead, write good clean code to start with, and refactor properly at regular intervals (which is the choice of the game's management, not necessarily the developers), then many of those problems can be avoided.
I know I'm doing a lot of monday morning quarterbacking with my comment, and I'm not disagreeing with the article, just food for thought.
QFT!
Here's her EQ rants btw. A worthy read as well. 18+ required :p
web.archive.org/web/20051217102848/http://tweety.bowlofmice.com/
After so many years doing community, I'm totally confused by kindness and approval. But thank you :)
Slushies - ah yes, my euphemism for "some of you people have driven me to drink, and it will take at least four shots of Jager to erase the memory of this day, but I can't talk directly about my incipient alcoholism on this website." Sweet memories!
@Vickykol - you, madam, assuming you are in fact a madam, are 100% correct and I wish I had a do-over for this article. I should have included those reasons. I personally play on Brandywine when I'm playing LOTRO because of those reasons, and would not transfer off for any incentive despite occasional queues and instability. But I also know that to keep my crackling chat channels, I'm gonna have to eat the lag. Maybe if everyone had to click something that said "This server is overloaded and occasionally crashes. Is it worth it to you? YES/NO." Because in most cases, the devs cannot do anything immediate about it, and can't give you an ETA on a fix at all.
It would help if dev people would pick a freaking engine suitable for their announced design, wouldn't it?
@Akousmata - what's it called if you're playing devil's advocate with someone widely considered to be Satan's minion? ;)
Okay, srsly - "write good clean code" - I have a followup next week from a technical director that goes into this a bit, so let me know what you think then.
@Bel: hahahahahahahahahaha
I tell people that your sins follow you forever on the internet, and here you are proving it!
I don't regret most of those. Some were me being young and ignorant. Some were me being drunk and maudlin. All of them were funny at the time. Thanks for dredging them up for me, I lost the originals years ago.
This was a blog or a rant?
Sorry but if you feel the need to write an article saying that an IT company (and all mmo developers are an IT company) cares about its backbone, you're probably dumber than the people saying "this company doesn't care about server stability". Or your just ranting.. either way, this blog equates too.. Pepsi really cares about its taste!..
duh.
Tell me why companies refuse innovation..
Tell me why every game has the same basic functions..
Tell me why paid beta is the industry norm..
c'mon.. give me the real underbelly.
Did McQuaid really choose smack over Vanguard???? that's what i want to know!!!
I feel better now.
Read my edits.. NOthing personal but when i hear underbelly i want to hear the dark truth to why companies refuse innovation, release games too early and don't listen to their fans. This industry has become stagnant. If it doesn't wake up soon it will be nothing but a memory much like the internet bubble.
You left out my personal favorites...when the "server stability" issues aren't actually that at all, but some transmission problem with 3rd party ISPs that you have no control over :)
Anyways, nothing there that I didn't know...but not everyone has had the thrill of working in a major ISP and all the fun and joy that brings to life....Always good to see at least an attempt to educate the masses.
Copeland: Nah, man, it's cool. I was joking with you - see my first post in the thread.
And thank you for the helpful list of article suggestions. I'll do my best.
I read it.. seriously Jager over Wild Turkey??
Here i thought you were cute but there can be no future with someone that chooses Jager over Kentucky bourbon... :P
Yeah that's what i took from it...
R U really Queen of the Galactic Universe????
Sanya: This is a little Off-Topic, but I'm just wondering what your thoughts on Mythic's Warhammer are. Just curious as to your feelings on how the game turned out and what you think about it compared to DAoC.
The info displayed upon server selection has always been population and operational status. Perhaps if MMOs listed average ping times, population and operational status, players wanting a high pop will go to those and players who are into smaller groups and soloing and are allergic to server-side lag would choose the ones with the lowest pop and the higher avg ping.
Maybe this way the server load would be a bit more evenly spread upon initial selection. As a given server becomes more popular, players of various sub-types would go to the ones they favoured and the load would be more even.
we just need as many good reasons to go as there are to stay when evaluating those 3 factors.
I love them, whatever they are. It does explain why only your first rant shows up on eatingbees tho. Glad to have helped :D
Sometimes, one have to wonder what some dev are thinking about, they should at least hire one tech on a 24h schedule with a pager, cell phone fax etc in case something goes boom with a server. Was a nice read and I'm very happy to see you (read you) again, im an old time daoc player and was kinda wondering where you would pop-up again:p
Welcome Sanya!
Good article, it revealed an aspect of the industry I had not given much thought to. I look forward to your next article.
I really hate to be the bad guy, but I am sorry in advance.
This is a medicore piece colorfully decorated with grammatical antics to "unbore" a bland topic. There are alot of better topics concerning "MMOs behind the scenes" that could've been covered with the right amount of time invested in sorting out the facts from the fiction. Every sentence was basically a "captain obvious" moment. These are common sense facts that any MMO gamer who doesn't have his head up his butt already realized by using the next best invention since slice bread, "Google.com." The rest of the sheep will always scream "OMG the company doesn't care" due to a lack of intelligence, we've grown to accept this rather rowdy crowd of new MMO players and promptly introduce them to the ignore feature commonly found in every MMO Game now-a-days. And we move on...
I wish you luck in future endeavors.
Word. Game Companies are Responsible for their product. STOP MAKING EXCUSES.
Yes Sanya I remember you from the early days, Let me be the one voice here that says :
Stop Being An Apologist For Mediocrity .
I know they hired you to promote their side of the story but when the SERVER-side lags, freezes, crashes DONT BLAME THE PLAYER. Blame the f'king Corporation for being too cheap to spend the money to FIX (omg actually FIX something???!!!) the problem.
Mind if propose the following explainations I think you'd hear from Sanya?
1) By and large, you say, "Why no innovation?" Look at WoW. All 12 million subscriptions are NOT out looking for the next best thing. A vast majority aren't even on message forums of any kind. People like safe and familiar, even if it can get dull, whether you chose to admit it or not. That's why it's much easier to do small, incremental EVOLUTIONARY changes, not REVOLUTIONARY ones.
Coincidentally, INVESTORS like safe and familiar too. A simple equation:
Innovative = Risk (Unproven, Untested).
MMO's = Large dump trucks full of money to make and support.
Innovation + MMO's = Risk + Large dump trucks full of money to make and support
2) I'm assuming you mean controls or interface? That's what I think of when you say "Functions". Or do you mean "Game Mechanics"? If it's the first, then I'd say becauseby and large, the majority of people LIKE those functions.. Because they're familiar, proven, and accepted. See number 1. Have you ever played UO? If not, please do and try it for a few weeks.. Try to look past the graphics and just get a feel for the game. The functionality in UO can be VERY frustrating. WoW functionality evolved from UO functionality, and it's large steps in the right direction.
3) Seriously, you can't figure out why paid beta is the norm? I'm hoping your being facetious. Three standard answers here, in somewhat releveant order:
A) Money - The longer your developing the game, the longer your sucking the capital from your investors without proving you'll pay them back. That's risk, and it costs. See number 1.
B) No MMO is ever Complete, bug free, etc. Why? They're MASSIVE, complex systems with a mind-boggling number of "parts" that all have to work together in perfect harmony. They're made by people, so there will ALWAYS be bugs. Why are they never complete? Well... How long would you stick around in a game, particualrly an MMO that never added new content? That never changed?
C) Time. Considering factors A and B, you have to do what you can to please your investors and push a game out before the NEXT generation of games is already being developed, so they have to weigh the pros and cons in such a way that it answers one very simple question: How soon can we push out a game that people will pay to play and tolerate it's issues so that we can prove to our investors that it's a profitable venture, so they KEEP investing in it and we can keep improving it?
I'd think it'd be along those lines, anyway. My opinion.
And Sanya wasn't involved with Sigil or McQuaid at any point in time, I don't think. But I know a developer who worked when Vanguard was under his control once wrote an article about his experiences (Don't recall where) but yes, I'm fairly certain he had some serious drug issues.
It's about time someone stood up for the people who are trying to earn their paycheck, by keeping our favorite games afloat.
Also, Nice Spinal Tap reference!
Sanya is no corporate shill. Those days ended after Mythic. If you remember her from the "early" days, or followed her since, you'd probably get that vibe.
And because she will never say it, for one reason or another, let me be at least one of the voices here that says:
GET. REAL.
They have money so they MUST be able to fix it?! Seriously??! That's your epiphany? What the fuck do you think they're making here, Lego castles???!!
God damn.. STUPID! Don't you think if they COULD fix it, they WOULD? You think they want jagoff, mouth-breathing half-wits foaming at the mouth about their product in a public venue every time their framerate drops? Has it escaped your keen intellectual prowess that maybe, just maybe, there are issues outside of their control, or there are a metric ass-ton of considerations to factor in, such as your computer's specific hardware and software connection, internet connection, ISP's, millions-to-billions of calculations per second that must be made on every SERVER, all of which might be a little difficult to account for? Jesus, it probably takes less processing power and IO operataions to fly the damn space shuttle than it takes to run an MMO server for an hour.
Sweet jesus man, turn off the nerdrage and use your grey matter. Microsoft has far more money than ANY MMO company, and they can't even make a damn WORD PROCESSOR that always functions perfectly.
I love Sanya! Glad to see her here.
They told a lot of stories. Misrepresented a lot of things. Tons of cash. Failed results.
Thanks for printing some excellent reality. Certain things need to be said and ultimately those who don't want to see the truth but keep imagining it for what that game (and others) MIGHT have been will always have hurt feelings and lash back at the messenger.
It's been kind of dark in the MMO department for the last year with these offerings. Thank you for bringing a flashlight, Sanya.
Yes they care so much that they buy the cheapest server technology available on the market and in some cases it's old, used and dated.
The big pay to play franchises might be different.
Good read.
Though I wonder if it would be too much for an MMO developer to stick with "current" generation technology, and just make a great game. I mean, given the abundance of MMOs in the industry, the lessons from server stability issues have been taught time and time again.
Yet every MMO dev says, "OUR GAME WILL BE THE BEST EVER WITH THE NEWEST EVERYTHING!"
... Unimpressed.
Probably because it's good for the computer hardware industry overall as people scramble to buy new PC components. Or maybe it's good for the technology industry because people are pushing the envelope. But both industries will still move at the speed of a locomotive because people still need to figure out how to run Crysis at max settings with a super high widescreen resolution while having their AA turned all the way up with lighting and shadows and bloom effects and whatever. Bleeding edge technology works for single player or small scale multiplayer (max of say, 32?) games.
But in the realm of MMOs, players get the shaft, companies go belly up, and games get more and more mediocre as they try to reinvent the cash cow that is WoW.
StarCraft is 12 years old. And people still play it. Why? Because it's awesome.
If an MMO dev didn't shoot themself in the foot by trying to render every single tree as if trees have never been rendered before, then maybe technology would be able to cope with being able to process 500 players in the same place at the same time.
Reasons given by Sanya for poor stability ( my responses in red, closing in yellow ):
The server is at the top of the list of options, alphabetically. No matter how many gentle suggestions appear in the announcement box for new players to please for the love of god choose Zifferelle as their new home, new players wind up picking Abericious. In this scenario, Abericious and Berisia are the only unstable servers. There is no way to fix it besides mandatory server transfers.
The server has a recognizable name. LOTRO’s first two servers are “Arkenstone” and “Brandywine.” Guess which one is overloaded. Hint: It’s the one recognized by the most casual of Tolkein fans.
Easily solved by implementing a simple "cap" which removes the ability to create a character from a new account when that cap is reached. Not enough spaces left for you and all your buddies? Go to the next server with enough room. They could even put a number next to the server representing how many spots are available on any particular server.
The last patch put in a highly desirable feature, one that measurably increased sales. A feature cited by market research groups as THE reason to resubscribe, or extend a subscription, or recommend the game to a friend. In testing, all was well. In release, not so much. Either the test server didn’t simulate the load well enough, or the problems didn’t build up enough to be noticeable for ninety days – sixty days longer than the new feature sat on the test server. Whatever the problem, the studio is now between a rock and a hard place. Remove the feature and lose money, or fail to fix the problem and lose money. Most people choose to work frantically on fixing the problem, but are hesitant to promise when the issue would be fixed… because if it were an easy fix, the players would never have noticed the problem at all.
Again, easily fixed by developers taking the time to work on and test new patches/content/expansions BEFORE even telling the world that it's coming. No rushed release, no disappointments in promised content/features not being implemented, and a generally bug free implementation.
The game might be more than a few years old, on its tenth group of server programmers. The resulting layers of code would take a genius with infinite free time to puzzle out. If the game is old enough, the server programmers are the youngest, greenest, newest hires not considered experienced enough to work on the unannounced Future of Gaming product. So the person least able to deal with the legacy is the one who has to try. This would seem to contradict my assertion that server stability is all the company cares about, but it truly doesn’t. In this scenario, the server programmer is one of the only people left on the team at all.
If the responses to the two above scenarios were implemented and practiced, along with adding in the simple requirement that all programmers accurately note every change/addition they made next to their code like normal software programmers do ( or the good ones do, anyway ), it wouldn't matter how old the game was or who was programming it.
Fact of the matter is, everything Sanya presented as an excuse for the "poor techs" would be remedied by simple common sense, hiring of competent talent, and a tad bit of effort on the developers part. Although this seems to be the unpopular opinion here, I'm going to have to agree with Flummoxed and others who just saw someone making excuses. To me, this "article" is nothing more than a fluff opinion piece in defense of shoddy business management.
This is the type of "fix" that empties games. Start telling your customers where they can and can;t make characters and who they can and can't play with and many of them will rapidly find things they'd rather do than play your game. If you're going to control their behavior, you need to do it either invisibly or in such a way as they see it as "being given autonomy".
There's another factor that comes in for choosing a server, which I have heard many people mention over the years: how easy is the server name to remember and/or pronounce. Some games offer a dozen server names of which only one or two look like recognizable English words, which is just asking for trouble.
Personally, as a player whose playstyle nowadays is mainly duoing with some soloing, I tend to look for the low population servers. In a PvE game, this has way more benefits than disadvantages, because with a solo/duo/small group playstyle your prime concern on busy servers is big groups muscling you out of the way of content you want to do. Also, when you do want to group up, on a low population server people tend to be much less picky about who they'll take. Also, my experience is that low-pop servers are a lot more relaxed and friendly.
The big disadvantage of low-pop is that younever know when your server is going to vanish from the list to be merged with some other server you never heard of, leaving you stranded on some new, high-pop server full of angry, competetive, displaced/imposed-upon people.
In a PvP game like Warhammer, which we just started, choosing low-pop really isn't an option. we stuck with the server the game selected for us during the free trial and its both busy and slightly unstable. Also I couldn't tell you the name of it to win a bet.
Actually, this kind of FIX worked pretty well with Aion back when it released in Korea, and this kind of FIX is MANDATORY for ANY RVR game, something Mythic miserably failed to do. There are plenty of options for developers to avoid overpopulated servers, there are also plenty of options to avoid unbalanced servers. Biggest problem is the one making the calls often lacks the knowledge.
Great new column! I'm looking forward to reading more about the underbelly.
So bitter. It's called laying the foundation. Every thesis I have ever read begins much the same way. There's a reason for that. As for people who respond emotionally to frustration they're neither being stupid or lacking in intelligence. They're just behaving as most humans do when in an emotional state. Not everyone is the Buddha. The ones that annoy me are the ones who persist after they have had time to have thought it through, and not done so. Even with them it's simply a matter of education. You have to walk them through the terra incognito.
This is the type of "fix" that empties games. Start telling your customers where they can and can;t make characters and who they can and can't play with and many of them will rapidly find things they'd rather do than play your game. If you're going to control their behavior, you need to do it either invisibly or in such a way as they see it as "being given autonomy".
There's another factor that comes in for choosing a server, which I have heard many people mention over the years: how easy is the server name to remember and/or pronounce. Some games offer a dozen server names of which only one or two look like recognizable English words, which is just asking for trouble.
Personally, as a player whose playstyle nowadays is mainly duoing with some soloing, I tend to look for the low population servers. In a PvE game, this has way more benefits than disadvantages, because with a solo/duo/small group playstyle your prime concern on busy servers is big groups muscling you out of the way of content you want to do. Also, when you do want to group up, on a low population server people tend to be much less picky about who they'll take. Also, my experience is that low-pop servers are a lot more relaxed and friendly.
The big disadvantage of low-pop is that younever know when your server is going to vanish from the list to be merged with some other server you never heard of, leaving you stranded on some new, high-pop server full of angry, competetive, displaced/imposed-upon people.
In a PvP game like Warhammer, which we just started, choosing low-pop really isn't an option. we stuck with the server the game selected for us during the free trial and its both busy and slightly unstable. Also I couldn't tell you the name of it to win a bet.
If I may interject here...
Anyone remember the world pass system from FFXI? It's basically the same idea as this. They force you to join a particular server (undoubtedly one with a lower population for balancing purposes) unless you have a code from someone on another server and you can enter that and join the other world with your friends. I'm not sure if this has changed, but it was used when the game was first released and it worked well.
So, I think I have to agree with the "nay-sayers" to an extent. Some of the points made in the article seem like poor excuses for a lack of imagination regarding design. Other points are well made and to me pretty clear facts. I think in the end all you need to do is look at games that have had more success in this area (of server stability) and then look at the ones who have failed in this area and find the difference and branch out from there. That's a good, logical way to move forward in regards to fixing this issue in my opinion.
In conclusion, there's always a solution so to say things are impossible to fix is just about as ignorant as saying you can fix them right away. So start by looking at systems in games that have worked well, take the good ideas, throw out the ones that don't work and start building better and better foundations to eliminate these problems before they start. There will always be issues, but it's the way you solve them that you have control over.
I love when people use those phrases, especially when their 'solution' shows a clear lack of understanding of the scope of the problem and the factors involved.
The 'easy' fix of caps, especially during the first few months would seriously affect the ability for guilds and FAF to coordinate servers they get on. It also does little to help retain new players later who join later in the game and are being sent to a low pop server. One of the biggest questions you see posted here or any forum from prospective players is "what is the population like?" Players dont want to be kicked to the dead server. They want to be in the most packed one. They more than likely won't talk to anyone outside of LFG for their entire time there but that doesn't chang the fact that there is a higher chance of retention of they are starting on a 'bustling' server.
I love when people use those phrases, especially when their 'solution' shows a clear lack of understanding of the scope of the problem and the factors involved.
The 'easy' fix of caps, especially during the first few months would seriously affect the ability for guilds and FAF to coordinate servers they get on. It also does little to help retain new players later who join later in the game and are being sent to a low pop server. One of the biggest questions you see posted here or any forum from prospective players is "what is the population like?" Players dont want to be kicked to the dead server. They want to be in the most packed one. They more than likely won't talk to anyone outside of LFG for their entire time there but that doesn't chang the fact that there is a higher chance of retention of they are starting on a 'bustling' server.
While I am not a developer, 20 years of gaming does give me some "insight" into the "scope and factors" regarding the issue at hand.
If the servers are setup this way well in advance so that most know about it, there is little reason to bitch about it. As has already been pointed out FFXI uses a similar system and it has not impacted their sales or "retention" at all. If your game is good, those servers won't be "dead".
For one thing, you don't open the game with 40 damn servers. You open with 4, for example, with more at the ready to "plug-in" when required. When those servers reach cap, you have the next server on standby where it will then appear in the selection list when the previous servers are at least 90% of cap. Then the 6th server, etc., etc. A big part of server "deadness" comes from pushing out 20-30 servers at launch, which spreads everyone all over the damn place, and then causes the aggravation of server merges, which also gives a negative view of your game when people start with the "server merges after only 3 monhs". Look at AoC and WAR for prime examples.
And when I use the term "easy fix", I am not saying it's just write one line of code and then pop off to lunch. What i am saying is that there are solutions which are far easier to do from the beginning than the impact of trying to stabalize and optimize a game when you're already 6 months to 4 years into the games' life.
It's time developers stopped relying on the "it's like this with every game, blah, blah, blah.", and start planning stuff from the beginning. Most developers have gotten too damned used to "push it out and patch it later", rather than having a real plan of attack from the beginning. And yes, i will concede that publishers are a problem in this area. But developers can change that, as well. A game which is ready to publish will find a publisher. A publisher unwilling to negotiate with the developers, if the developers will start standing together in terms of making the games ready first, will find themselves out of business. While publishers willing to let the developers make the game correctly will have business hand over fist.
And I also know that developers are making the games first and fremost to make money. What they need to understand is if they start making the games correctly from the beginning, they'll make a whole lot more money in the short term as well as the long term when they are having to only spend money for ongoing development, rather than to fix crap they should have fixed before launch while also having to develop content to keep the customers they have, as well as the money not only saved but earned by not having to worry about 20k beta testers on forums everywhere letting people know not to play the buggy POS.
great write up. love articles like this and can't wait till the next one.
Haha! Sanya's back! effin ay i miss the old days of the Grab Bag for DAoC. Back when the game wasn't killed by that infernal creation that blizzard released upon the world. DAMN YOU BLIZZARD!
Sanya, so... what was the reason ya left soon after EA acquired Mythic? it was too close togethor to not be noticeable XD You can tell me... i'll keep it a secret D:
Continue publishing articles and columns like this and less viral marketing for companies like SOE and this site may become relevant again.
not sure if this has been posted or not but therre is a slight mistake in the big image link to this article on the front page
"Sanya Weathers, the former Director of Community Relations for Mythic Entertainment, brings her unique voice to MMORPG.com in his new column. In her debut, she looks behind the scenes at server stability. "
Hello Sany and thanks for the article.
As a former IT admin, then manager I agree with the statement that for IT companies, applications availability server stability are the #1 concerns.
As pointed by someone it might looks like saying taste is important for Pepsi but well not everyone is aware of this. (btw brand image might even be more important than taste for Pepsi and Coke... )
So from all IT related people, I'd like to say thank you for pointing this out. I'd like also to add these people are, too often, spending well deserved sleep hours working to fix things.
Regarding your statement "The last patch put in a highly desirable feature" I was wondering what your were sepaking about.
It applies to any patch for any MMO, really. Do you pull the broken feature that players have been waiting for, or leave it in broken while you work feverishly to fix it. Either way, a great unhappiness ensues.
It applies to any patch for any MMO, really. Do you pull the broken feature that players have been waiting for, or leave it in broken while you work feverishly to fix it. Either way, a great unhappiness ensues.
Oh! thanks for the explanation.
There is only one answer, once it's done leave it there and even improve it. Customers and players alike don't care how difficult it's to maintain your environement. You'd had to think about it befrore implementing it.
"Oh! thanks for the explanation.
There is only one answer, once it's done leave it there and even improve it. Customers and players alike don't care how difficult it's to maintain your environement. You'd had to think about it befrore implementing it."
you didn't really read this article at all did you? It's basically saying you need to realize that they TRY very hard to think of every possible scenario so something works as intended. But not everything is predictable.
There are too many posts for me to quote directly, so I will not. If you have been reading the comments I am sure you will recognise to whom I am referring.
Businesses that plan new software will always start with good intentions, so the planing and design phases are usually the best part. Unfortunately, everyone that wants the software, both the financiers and the users, keep pushing and pushing to get it earlier. This is where the problems start. Ever heard the feedback from code jockeys about what happens in the closing stages of MMO development? More and more people get hired to cut the code faster and faster. Pressure is applied to keep working longer and longer hours (for no extra compensation other than to state on your Resume that you were there). More pressure and less sleep means more minor mistakes that will haunt the venture later.
I recall when a major mainframe company planned a completely revamped version of their operating system (think DOS to Windows type upgrade but for mainframes). They spent millions of dollars and many years planning and designing it to prevent all the problems that were common place in their original software. Unfortunately, pressure was being constantly applied to get it out to the market so it be charged for and R&D costs recovered. As a result, someone decided that the fastest and cheapest way to cut the code was to hire the entire 1st year IT student body at the local university. After all, the design had been done properly and how difficult is it to just cut code?
Are MMOs more complex than software systems in the space shuttle? Yes and no. If you equate "Game Mechanics" to the interaction of multiple systems keeping people alive in space, then the shuttle wins by orders of magnitude. What does make MMOs more complex is everything that drives the graphics systems. Today's average PC is many 1000's of times more powerful than the first IBM PC, yet only a handful of people worldwide could type faster than that original 4.77MHz CPU could handle. What are we using all that extra processing power for? 90% of it is chewed up in graphics.
I freely admit that I love seeing how beautiful some of these new MMO worlds have been made. But if someone offered to upgrade my WoW graphics to "Real Life", full 3D, wear a body suit and goggles to immerse yourself in the game, at the cost of never getting past 4 fps, I would not take it. I would love to see it ... once ... but I would not game in that environment. I think part of the reason game companies plan on using the cutting edge technology is the belief that once the game goes live, what was cutting edge when the plan started will be common on most PCs. They need to take a step back from that belief.
What do I see for the future? The mistakes being made today by today's MMO companies are the same ones made many years and decades ago in industries all over the world. They are the same mistakes that the civil engineering industry made in the 1950's and 60's and the same that have resulted in the current world-wide financial system crisis. Boards of Directors taking their eyes off the actual product they make (or service they deliver) to push some artifical ranking number (eg share price) a little higher just so they can claim to be #1. When you forget about what you are doing, you stop doing it well and if that keeps up, you will not be doing it for long.
Sorry folks, but I do not see it ever getting better.
I have no doubt that the people whose job it is to keep the servers running almost invariably care very much about keeping the servers running smoothly. But that's not what people mean when they claim that a company doesn't care about server stability. If the problem is that the company doesn't have enough hardware, or good enough hardware, or won't hire enough people to run it properly, and that problem persists for a long period of time, then the company doesn't care about server stability. Or perhaps rather, the company cares about other things more than server stability, and when they have to make choices, are fine with letting the servers be a mess.
I can understand that there will be problems. If a server crashes now and then, or even has to be offline for several hours during prime time because something unforeseeable went wrong, I can understand that. If a game runs fine under normal circumstances, but chokes when you try to get 50 people in an area at once, I can understand that, too--unless the game creates incentives to cram as many people into an area as possible. What I'm not so understanding of is when servers are unstable day after day for months at a time without the company fixing it.
That may sound rather harsh; most companies I've seen do a respectable job of keeping their servers up and running. I've only seen two exceptions to this. One was the obscure 2am Games, which went out of business because, well, they really needed to go out of business. It would have been a severe flaw of capitalism if it couldn't put 2am Games out of business. The other exception is Blizzard, whose server problems, while inexcusably bad, were far shy of what 2am did, and Blizzard also did a lot of other things well.
A number of things in the article seem to be saying, "the servers are down, but it's not our fault, and there's nothing we can do about it". That may be true in the sense that it's not the fault of the people whose job is to run the servers. But that doesn't mean it's not the company's fault. If a company comes out and says, we're never going to have another patch to fix bugs, that wouldn't be the fault of the people who run the servers, either, but it would very much be the fault of the company--and it would be fair to say that a company that did so didn't care about fixing bugs.
The article talks at length about the order of servers leading to some having far more players than others. The overcrowded servers thus have lag problems. Where did the order of servers on the list come from? Was it forced upon you by the nature of the universe? Or did someone at the company have to code it as such? If a bad way of listing the servers available funnels too many players into some and not enough into others, that's the fault of the company.
I can understand that some bugs are very, very hard to track down and fix. If all a player can tell you is that the game is crashing, there are many, many things that could potentially cause that. It could be subtle interactions of disparate code pieces written by different people. I get that. But you can't find the code that sorts the servers on a list? Really? Carelessness there may not be the fault of those whose job is to run the servers, but it is very much the fault of the company. A company that takes no measures to address that can fairly be said not to care about the problem.
Perhaps more fundamental is the assumption that there need to be separate servers with players prohibited from moving from one to another in the first place. Wizard101 has no such problems, for example. Players can freely switch from one server to another almost whenever they feel like it (60 second delay, not available when in an instance). Guild Wars likewise has no problems with one server being too crowded and another too empty. A company that chose to go the rigidly separate servers route chose all the problems that arise from that model. Whatever problems arise from that choice are the fault of the company.
There's this magical thing we IT people call "virtualization" that fixes almost all those issues... Get with the new technology and consolidate resources.
Nice artickle but I do understand why most WAR players pick up the servers that already have most players (and the LOTRO players for that matter).
The game is far more fun when there are more people around, there are a lot of discussions here about which server that have most population so people can choose them.
And as I see it, the dev must cap the server so it cant get too many players. In this case the problem came out of nowhere but there are more discussions about lagg here so I think Mythic have taken water over their heads and let in too many players on the servers, either they will have to put some kind of tech wizzard to make it all possible or lower the number of players (they dont have to kick anyone, just dont let any new players up until enough quited).
A PvP game with lagg is the worst possibly game to play. I can uderstand that crap just happens but Mythic must fix this ASAP or lose a lot of subs. No need to point fingers whos fault things are.
I like pictures
I did. I was adding some emplasis on her post.
I did. I was adding some emplasis on her post.
Originally posted by fatchickenj
There's this magical thing we IT people call "virtualization" that fixes almost all those issues... Get with the new technology and consolidate resources.
Yes and no. Virtualization can only go as far as your architecture let you consolidate things.
So if it wasn't planned beforehand it won't help you at all. Not every architecture is scalable and no matter what you do you'll have things like I/O bottlenecks, latency issues. Lately we virtualized a farm of 300+ Unix/Linux/Windows severs and well Virtualization can be your worst ennemy.
ROFL, I'm only an elite member for all the BS I spew here, nothing more. Dismissing my entire post for one off hand remark is being a bit sensitive don't you think? (and many F2P's are not well run, IMO at least)
As for being a troll, its a title I proudly wear.
Ok so basically whats shes saying in a underlying tone that Turbine is over subscribing servers, and dont have anyone working on real server optimizations from within the code anymore because its obvious that part of the code is "good enough" so no real need to keep up on that part anymore, which in turn means more money in the pocket of the executives whom were former AIG employees that decied that profits is more important than keeping on at least 2 people to keep server code up to date and more improved after each new addition of content which is supposed to be their 1st priority with the game, and should be with all the millions of dollars theyre making off the game during a economnic crunch which I might add isnt hurting the gaming industry so much as its more or less a excuse to rocket sales even further than before by laying thousands of people who were working for less than meager wages to start with, bringing around the question does Turbine really do "care" about their game, and their customer base....(Takes a huge breath...wipes brow!)........right?
To me in a "128 bit Sanya encrypted nutshell" she is pretty much saying the exact same thing that the gamer she was ridiculing about the company not "caring" all because the gamer didnt break it down in a pie and relabeled it BS? Wow Sanya I sorta thought that when someone saves someone from having to hear someone talking BS and trying to make it look prettier than a pig with a silk purse on, was doing me, and a few billion intelligent people on the planet a favor....(take another huge breath)....right?
Now I see why Mythic picked you up, but you cant BS a BSer Sanya, but +1 to you for trying! Have a awesome "non-BS" day!
Me either.
And that makes me very sad.
From my observation, "the industry" is so protective of itself that it doesn't let new people with new ideas in unless they have suchandsuch a degree beneath years and years already in the industry. Then those people have to have a game that is somewhat made, and not just a design document. THEN, those people have to hope their game isn't "too risky" so that someone with money will invest in it.
All the while, we have to hope that the industry hasn't warped such a potentially bright mind into not exploring the boundaries of video game design and instead looking for the next cash cow based on doing what other games have done previously. Which, essentially... is fail.
The industry will, essentially, choke itself.
All while thinking they're wonderful... and ignoring the fact that they fail to deliver time and time and time and time again.
I'd like to punch a few CEOs in the face to be honest.
Well said and I hope it eleviates some of the typical flamers I see when lotro has to bring servers down for hot fixes. Of course I know better too.
Excellent read, and as a Sys-Admin myself, well-appreciated. I understand, fully, having to wake up in the middle of the night to resolve server issues. Not for an MMO, mind you, but server issues are server issues.
The biggest problem with MMOs is the whole "dude, what server are you on, I'll join that one". Most MMOs require you to limit yourself to one server, or shard, so that you can join up with others you know. The fall-back from this, of course, is the more popular a server is, the heavier the load. The heavier the load, the more issues the end-user will see.
MMOs and the sharded world most rely on have always experienced this problem. Players want to be part of the most action-packed server. The most action-packed servers, however, will eventually not be much fun to play on once they reach max-capacity.
I think one thing MMO vendors could do is make server transfers free. Not free forever, mind you, but free maybe once every 6 months. This would make it more attractive for players who dislike server lag to pickup and move to a lighter server, or even pick up and move an entire guild to a lighter server for that matter. This would be advantageous to both players and the vendor.
That's silly. There is no single more important aspect to a MMO than its servers; if they don't work, there's no game any more, and unavailability is a good way to get people to try and get hooked on your competition.
Now you have a half-point in that like many services MMOs do 'oversell' in that they are not providing capacity for every single subscriber to be online at one time, but then you throw out your half-point by venturing into hyperbole.
Besides, if you had been looking, you would have noticed that lately Turbine was in fact advertising for a server programmer. I know this because I was recently in the job hunt and almost applied, but MA is a bit more uprooting than I wanted to do at the moment.
Originally posted by Sanya
Next week, I’ve got some discussion with an actual server professional to share with you. Tune in then!
Oh, I can't wait to see who that might be... could it be her long-time friend and one-time Mythic server professional Lum? What are the odds?
There are many important aspects for any project, MMO included.
But the original article does hit the nail, if the server flops, all designa and vision are gone. People can stare at your website, the artwork of your boxset, and listen to the musical score (when the screen says, "server down") for just so long.
The server has to sustain the game. Game design has to be implemented and that is where the server comes in. As a designer at times, I know how many times my design is compromised b/c the engineer need to implement it in a budge limit. Can't blame him either, he need to meet the budget or he is fired.
During the development stage, everything is a compromise, the lore wants this, the designer has to design a gameplay that is playable, not just loyal to lore. The engineer has to implement it, not just loyal to design, but limited to budget. The programmer must implement the engineer's (read system design) wishes according to the limits of the platform and engine.
Everything is a compromise, from lore to design to implementation to coding. Budget, time constraint and "vision" about the future platform on which the game (or whatever system being developed) is to be delivered on. Blame no one in particular. Its a team work, in which everyone want his own but ends up compromising.
Hey Sanya Weathers!!! I'm a 24 year old cpa in the process of getting his actuarial degree. I'm also a male model with a great body and above average face. Let me know if you want to go out sometime. Nice article too....8474146382
That's silly. There is no single more important aspect to a MMO than its servers; if they don't work, there's no game any more, and unavailability is a good way to get people to try and get hooked on your competition.
Now you have a half-point in that like many services MMOs do 'oversell' in that they are not providing capacity for every single subscriber to be online at one time, but then you throw out your half-point by venturing into hyperbole.
Besides, if you had been looking, you would have noticed that lately Turbine was in fact advertising for a server programmer. I know this because I was recently in the job hunt and almost applied, but MA is a bit more uprooting than I wanted to do at the moment.
Originally posted by Sanya
Next week, I’ve got some discussion with an actual server professional to share with you. Tune in then!
Oh, I can't wait to see who that might be... could it be her long-time friend and one-time Mythic server professional Lum? What are the odds?
Wow, did someone hit you over the head with a hammer? Try to take my WHOLE post into context instead of a small section that can be interpreted into anything ya want. F`ing liberal tree huggers, give them books, and give them books, and all they do is eat the pages!