Originally posted by WisebutCruel

Reasons given by Sanya for poor stability ( my responses in red, closing in yellow ):

The server is at the top of the list of options, alphabetically. No matter how many gentle suggestions appear in the announcement box for new players to please for the love of god choose Zifferelle as their new home, new players wind up picking Abericious. In this scenario, Abericious and Berisia are the only unstable servers. There is no way to fix it besides mandatory server transfers.
The server has a recognizable name. LOTRO’s first two servers are “Arkenstone” and “Brandywine.” Guess which one is overloaded. Hint: It’s the one recognized by the most casual of Tolkein fans.

Easily solved by implementing a simple "cap" which removes the ability to create a character from a new account when that cap is reached. Not enough spaces left for you and all your buddies? Go to the next server with enough room. They could even put a number next to the server representing how many spots are available on any particular server.

This is the type of "fix" that empties games. Start telling your customers where they can and can;t make characters and who they can and can't play with and many of them will rapidly find things they'd rather do than play your game. If you're going to control their behavior, you need to do it either invisibly or in such a way as they see it as "being given autonomy".

There's another factor that comes in for choosing a server, which I have heard many people mention over the years: how easy is the server name to remember and/or pronounce. Some games offer a dozen server names of which only one or two look like recognizable English words, which is just asking for trouble.

Personally, as a player whose playstyle nowadays is mainly duoing with some soloing, I tend to look for the low population servers. In a PvE game, this has way more benefits than disadvantages, because with a solo/duo/small group playstyle your prime concern on busy servers is big groups muscling you out of the way of content you want to do. Also, when you do want to group up, on a low population server people tend to be much less picky about who they'll take. Also, my experience is that low-pop servers are a lot more relaxed and friendly.

The big disadvantage of low-pop is that younever know when your server is going to vanish from the list to be merged with some other server you never heard of, leaving you stranded on some new, high-pop server full of angry, competetive, displaced/imposed-upon people.

In a PvP game like Warhammer, which we just started, choosing low-pop really isn't an option. we stuck with the server the game selected for us during the free trial and its both busy and slightly unstable. Also I couldn't tell you the name of it to win a bet.

Actually, this kind of FIX worked pretty well with Aion back  when it released in Korea, and this kind of FIX is MANDATORY for ANY RVR game, something Mythic miserably failed to do. There are plenty of options for developers to avoid overpopulated servers, there are also plenty of options to avoid unbalanced servers. Biggest problem is the one making the calls often lacks the knowledge. 

Great new column!  I'm looking forward to reading more about the underbelly.

Originally posted by Nevulus

I really hate to be the bad guy, but I am sorry in advance.

This is a medicore piece colorfully decorated with grammatical antics to "unbore" a bland topic. There are alot of better topics concerning "MMOs behind the scenes" that could've been covered with the right amount of time invested in sorting out the facts from the fiction. Every sentence was basically a "captain obvious" moment. These are common sense facts that any MMO gamer who doesn't have his head up his butt already realized by using the next best invention since slice bread, "Google.com." The rest of the sheep will always scream "OMG the company doesn't care" due to a lack of intelligence, we've grown to accept this rather rowdy crowd of new MMO players and promptly introduce them to the ignore feature commonly found in every MMO Game now-a-days. And we move on...

 

I wish you luck in future endeavors.

So bitter. It's called laying the foundation. Every thesis I have ever read begins much the same way. There's a reason for that. As for people who respond emotionally to frustration they're neither being stupid or lacking in intelligence. They're just behaving as most humans do when in an emotional state. Not everyone is the Buddha. The ones that annoy me are the ones who persist after they have had time to have thought it through, and not done so. Even with them it's simply a matter of education. You have to walk them through the terra incognito.

Originally posted by Bhagpuss

Originally posted by WisebutCruel

Reasons given by Sanya for poor stability ( my responses in red, closing in yellow ):

The server is at the top of the list of options, alphabetically. No matter how many gentle suggestions appear in the announcement box for new players to please for the love of god choose Zifferelle as their new home, new players wind up picking Abericious. In this scenario, Abericious and Berisia are the only unstable servers. There is no way to fix it besides mandatory server transfers.
The server has a recognizable name. LOTRO’s first two servers are “Arkenstone” and “Brandywine.” Guess which one is overloaded. Hint: It’s the one recognized by the most casual of Tolkein fans.

Easily solved by implementing a simple "cap" which removes the ability to create a character from a new account when that cap is reached. Not enough spaces left for you and all your buddies? Go to the next server with enough room. They could even put a number next to the server representing how many spots are available on any particular server.

 

This is the type of "fix" that empties games. Start telling your customers where they can and can;t make characters and who they can and can't play with and many of them will rapidly find things they'd rather do than play your game. If you're going to control their behavior, you need to do it either invisibly or in such a way as they see it as "being given autonomy".

There's another factor that comes in for choosing a server, which I have heard many people mention over the years: how easy is the server name to remember and/or pronounce. Some games offer a dozen server names of which only one or two look like recognizable English words, which is just asking for trouble.

Personally, as a player whose playstyle nowadays is mainly duoing with some soloing, I tend to look for the low population servers. In a PvE game, this has way more benefits than disadvantages, because with a solo/duo/small group playstyle your prime concern on busy servers is big groups muscling you out of the way of content you want to do. Also, when you do want to group up, on a low population server people tend to be much less picky about who they'll take. Also, my experience is that low-pop servers are a lot more relaxed and friendly.

The big disadvantage of low-pop is that younever know when your server is going to vanish from the list to be merged with some other server you never heard of, leaving you stranded on some new, high-pop server full of angry, competetive, displaced/imposed-upon people.

In a PvP game like Warhammer, which we just started, choosing low-pop really isn't an option. we stuck with the server the game selected for us during the free trial and its both busy and slightly unstable. Also I couldn't tell you the name of it to win a bet.

If I may interject here...

Anyone remember the world pass system from FFXI? It's basically the same idea as this. They force you to join a particular server (undoubtedly one with a lower population for balancing purposes) unless you have a code from someone on another server and you can enter that and join the other world with your friends. I'm not sure if this has changed, but it was used when the game was first released and it worked well.

So, I think I have to agree with the "nay-sayers" to an extent. Some of the points made in the article seem like poor excuses for a lack of imagination regarding design. Other points are well made and to me pretty clear facts. I think in the end all you need to do is look at games that have had more success in this area (of server stability) and then look at the ones who have failed in this area and find the difference and branch out from there. That's a good, logical way to move forward in regards to fixing this issue in my opinion.

In conclusion, there's always a solution so to say things are impossible to fix is just about as ignorant as saying you can fix them right away. So start by looking at systems in games that have worked well, take the good ideas, throw out the ones that don't work and start building better and better foundations to eliminate these problems before they start. There will always be issues, but it's the way you solve them that you have control over.

Originally posted by WisebutCruel

Easily solved by...

Again, easily fixed by...

I love when people use those phrases, especially when their 'solution' shows a clear lack of understanding of the scope of the problem and the factors involved.

The 'easy' fix of caps, especially during the first few months would seriously affect the ability for guilds and FAF to coordinate servers they get on. It also does little to help retain new players later who join later in the game and are being sent to a low pop server. One of the biggest questions you see posted here or any forum from prospective players is "what is the population like?" Players dont want to be kicked to the dead server. They want to be in the most packed one. They more than likely won't talk to anyone outside of LFG for their entire time there but that doesn't chang the fact that there is a higher chance of retention of they are starting on a 'bustling' server.

Originally posted by LynxJSA

Originally posted by WisebutCruel

Easily solved by...

Again, easily fixed by...

 

I love when people use those phrases, especially when their 'solution' shows a clear lack of understanding of the scope of the problem and the factors involved.

 

The 'easy' fix of caps, especially during the first few months would seriously affect the ability for guilds and FAF to coordinate servers they get on. It also does little to help retain new players later who join later in the game and are being sent to a low pop server. One of the biggest questions you see posted here or any forum from prospective players is "what is the population like?" Players dont want to be kicked to the dead server. They want to be in the most packed one. They more than likely won't talk to anyone outside of LFG for their entire time there but that doesn't chang the fact that there is a higher chance of retention of they are starting on a 'bustling' server.

 

 While I am not a developer, 20 years of gaming does give me some "insight" into the "scope and factors" regarding the issue at hand.

If the servers are setup this way well in advance so that most know about it, there is little reason to bitch about it. As has already been pointed out FFXI uses a similar system and it has not impacted their sales or "retention" at all. If your game is good, those servers won't be "dead".

For one thing, you don't open the game with 40 damn servers. You open with 4, for example, with more at the ready to "plug-in" when required. When those servers reach cap, you have the next server on standby where it will then appear in the selection list when the previous servers are at least 90% of cap. Then the 6th server, etc., etc. A big part of server "deadness" comes from pushing out 20-30 servers at launch, which spreads everyone all over the damn place, and then causes the aggravation of server merges, which also gives a negative view of your game when people start with the "server merges after only 3 monhs". Look at AoC and WAR for prime examples.

And when I use the term "easy fix", I am not saying it's just write one line of code and then pop off to lunch. What i am saying is that there are solutions which are far easier to do from the beginning than the impact of trying to stabalize and optimize a game when you're already 6 months to 4 years into the games' life.

It's time developers stopped relying on the "it's like this with every game, blah, blah, blah.", and start planning stuff from the beginning. Most developers have gotten too damned used to "push it out and patch it later", rather than having a real plan of attack from the beginning. And yes, i will concede that publishers are a problem in this area. But developers can change that, as well.  A game which is ready to publish will find a publisher. A publisher unwilling to negotiate with the developers, if the developers will start standing together in terms of making the games ready first, will find themselves out of business. While publishers willing to let the developers make the game correctly will have business hand over fist.

And I also know that developers are making the games first and fremost to make money. What they need to understand is if they start making the games correctly from the beginning, they'll make a whole lot more money in the short term as well as the long term when they are having to only spend money for ongoing development, rather than to fix crap they should have fixed before launch while also having to develop content to keep the customers they have, as well as the money not only saved but earned by not having to worry about 20k beta testers on forums everywhere letting people know not to play the buggy POS.

great write up. love articles like this and can't wait till the next one.

Haha! Sanya's back! effin ay i miss the old days of the Grab Bag for DAoC. Back when the game wasn't killed by that infernal creation that blizzard released upon the world. DAMN YOU BLIZZARD!

Sanya, so... what was the reason ya left soon after EA acquired Mythic? it was too close togethor to not be noticeable XD You can tell me... i'll keep it a secret D:

Continue publishing articles and columns like this and less viral marketing for companies like SOE and this site may become relevant again.

not sure if this has been posted or not but therre is a slight mistake in the big image link to this article on the front page

"Sanya Weathers, the former Director of Community Relations for Mythic Entertainment, brings her unique voice to MMORPG.com in his new column. In her debut, she looks behind the scenes at server stability. "

 Hello Sany and thanks for the article.

As a former IT admin, then manager I agree with the statement that for IT companies, applications availability server stability are the #1 concerns.

As pointed by someone it might looks like saying taste is important for Pepsi but well not everyone is aware of this. (btw brand image might even be more important than taste for Pepsi and Coke... )

So from all IT related people, I'd like to say thank you for pointing this out. I'd like also to add these people are, too often, spending well deserved sleep hours working to fix things.

Regarding your statement "The last patch put in a highly desirable feature" I was wondering what your were sepaking about.

Originally posted by Deewe

Regarding your statement "The last patch put in a highly desirable feature" I was wondering what your were sepaking about.

It applies to any patch for any MMO, really.  Do you pull the broken feature that players have been waiting for, or leave it in broken while you work feverishly to fix it.  Either way, a great unhappiness ensues.

Originally posted by LynxJSA

Originally posted by Deewe

Regarding your statement "The last patch put in a highly desirable feature" I was wondering what your were sepaking about.

 

It applies to any patch for any MMO, really.  Do you pull the broken feature that players have been waiting for, or leave it in broken while you work feverishly to fix it.  Either way, a great unhappiness ensues.

 

 

Oh! thanks for the explanation.

There is only one answer, once it's done leave it there and even improve it. Customers and players alike don't care how difficult it's to maintain your environement. You'd had to think about it befrore implementing it.

"Oh! thanks for the explanation.

There is only one answer, once it's done leave it there and even improve it. Customers and players alike don't care how difficult it's to maintain your environement. You'd had to think about it befrore implementing it."

you didn't really read this article at all did you? It's basically saying you need to realize that they TRY very hard to think of every possible scenario so something works as intended. But not everything is predictable.

There are too many posts for me to quote directly, so I will not.  If you have been reading the comments I am sure you will recognise to whom I am referring.

Businesses that plan new software will always start with good intentions, so the planing and design phases are usually the best part.  Unfortunately, everyone that wants the software, both the financiers and the users, keep pushing and pushing to get it earlier.  This is where the problems start.  Ever heard the feedback from code jockeys about what happens in the closing stages of MMO development?  More and more people get hired to cut the code faster and faster.  Pressure is applied to keep working longer and longer hours (for no extra compensation other than to state on your Resume that you were there).  More pressure and less sleep means more minor mistakes that will haunt the venture later. 

I recall when a major mainframe company planned a completely revamped version of their operating system (think DOS to Windows type upgrade but for mainframes).  They spent millions of dollars and many years planning and designing it to prevent all the problems that were common place in their original software.  Unfortunately, pressure was being constantly applied to get it out to the market so it be charged for and R&D costs recovered.  As a result, someone decided that the fastest and cheapest way to cut the code was to hire the entire 1st year IT student body at the local university.  After all, the design had been done properly and how difficult is it to just cut code?

Are MMOs more complex than software systems in the space shuttle?  Yes and no.  If you equate "Game Mechanics" to the interaction of multiple systems keeping people alive in space, then the shuttle wins by orders of magnitude.  What does make MMOs more complex is everything that drives the graphics systems.  Today's average PC is many 1000's of times more powerful than the first IBM PC, yet only a handful of people worldwide could type faster than that original 4.77MHz CPU could handle.  What are we using all that extra processing power for?  90% of it is chewed up in graphics.

I freely admit that I love seeing how beautiful some of these new MMO worlds have been made.  But if someone offered to upgrade my WoW graphics to "Real Life", full 3D, wear a body suit and goggles to immerse yourself in the game, at the cost of never getting past 4 fps, I would not take it.  I would love to see it ... once ... but I would not game in that environment.  I think part of the reason game companies plan on using the cutting edge technology is the belief that once the game goes live, what was cutting edge when the plan started will be common on most PCs.  They need to take a step back from that belief.

What do I see for the future?  The mistakes being made today by today's MMO companies are the same ones made many years and decades ago in industries all over the world.  They are the same mistakes that the civil engineering industry made in the 1950's and 60's and the same that have resulted in the current world-wide financial system crisis.  Boards of Directors taking their eyes off the actual product they make (or service they deliver) to push some artifical ranking number (eg share price) a little higher just so they can claim to be #1.  When you forget about what you are doing, you stop doing it well and if that keeps up, you will not be doing it for long.

Sorry folks, but I do not see it ever getting better.

I have no doubt that the people whose job it is to keep the servers running almost invariably care very much about keeping the servers running smoothly.  But that's not what people mean when they claim that a company doesn't care about server stability.  If the problem is that the company doesn't have enough hardware, or good enough hardware, or won't hire enough people to run it properly, and that problem persists for a long period of time, then the company doesn't care about server stability.  Or perhaps rather, the company cares about other things more than server stability, and when they have to make choices, are fine with letting the servers be a mess.

I can understand that there will be problems.  If a server crashes now and then, or even has to be offline for several hours during prime time because something unforeseeable went wrong, I can understand that.  If a game runs fine under normal circumstances, but chokes when you try to get 50 people in an area at once, I can understand that, too--unless the game creates incentives to cram as many people into an area as possible.  What I'm not so understanding of is when servers are unstable day after day for months at a time without the company fixing it.

That may sound rather harsh; most companies I've seen do a respectable job of keeping their servers up and running.  I've only seen two exceptions to this.  One was the obscure 2am Games, which went out of business because, well, they really needed to go out of business.  It would have been a severe flaw of capitalism if it couldn't put 2am Games out of business.  The other exception is Blizzard, whose server problems, while inexcusably bad, were far shy of what 2am did, and Blizzard also did a lot of other things well.

A number of things in the article seem to be saying, "the servers are down, but it's not our fault, and there's nothing we can do about it".  That may be true in the sense that it's not the fault of the people whose job is to run the servers.  But that doesn't mean it's not the company's fault.  If a company comes out and says, we're never going to have another patch to fix bugs, that wouldn't be the fault of the people who run the servers, either, but it would very much be the fault of the company--and it would be fair to say that a company that did so didn't care about fixing bugs.

The article talks at length about the order of servers leading to some having far more players than others.  The overcrowded servers thus have lag problems.  Where did the order of servers on the list come from?  Was it forced upon you by the nature of the universe?  Or did someone at the company have to code it as such?  If a bad way of listing the servers available funnels too many players into some and not enough into others, that's the fault of the company.

I can understand that some bugs are very, very hard to track down and fix.  If all a player can tell you is that the game is crashing, there are many, many things that could potentially cause that.  It could be subtle interactions of disparate code pieces written by different people.  I get that.  But you can't find the code that sorts the servers on a list?  Really?  Carelessness there may not be the fault of those whose job is to run the servers, but it is very much the fault of the company.  A company that takes no measures to address that can fairly be said not to care about the problem.

Perhaps more fundamental is the assumption that there need to be separate servers with players prohibited from moving from one to another in the first place.  Wizard101 has no such problems, for example.  Players can freely switch from one server to another almost whenever they feel like it (60 second delay, not available when in an instance).  Guild Wars likewise has no problems with one server being too crowded and another too empty.  A company that chose to go the rigidly separate servers route chose all the problems that arise from that model.  Whatever problems arise from that choice are the fault of the company.

There's this magical thing we IT people call "virtualization" that fixes almost all those issues... Get with the new technology and consolidate resources.
 

Nice artickle but I do understand why most WAR players pick up the servers that already have most players (and the LOTRO players for that matter).

The game is far more fun when there are more people around, there are a lot of discussions here about which server that have most population so people can choose them.

And as I see it, the dev must cap the server so it cant get too many players. In this case the problem came out of nowhere but there are more discussions about lagg here so I think Mythic have taken water over their heads and let in too many players on the servers, either they will have to put some kind of tech wizzard to make it all possible or lower the number of players (they dont have to kick anyone, just dont let any new players up until enough quited).

A PvP game with lagg is the worst possibly game to play. I can uderstand that crap just happens but Mythic must fix this ASAP or lose a lot of subs. No need to point fingers whos fault things are.

I like pictures