Sanya Weathers's MMO Underbelly: Tech Talk with Andrew
Each Friday, the former Director of Community Relations for Mythic Entertainment pens this column that focuses on some under-served MMO story. In this installment, she tells relays a conversation with server guy Andrew Mann talking about some of the issues behind the technology of MMOs.
For last week's article about server stability, I asked some questions of Andrew Mann, an old friend whose job it is (well, among many of his jobs) to keep the servers running at a major MMO studio. Of course, he gave me so much information that our chat became a feature of its own. I must state up front that the material in quotes is from Andrew. The material not in quotes consists of my own thoughts and observations. You'd think that would go without saying, wouldn't you? Ha! But that's a different topic entirely. Read on for an old insider's look at stability bugs and more.
The first thing I asked him was to give me a little behind the scenes peek at how stability problems get solved.
He said, "Stability problems start out as bugs, but turn into prioritization. When you find a bug that affects gameplay, you look at the level of impact on the players - how often, how many players, and how badly their experience is impacted. You compare that against an educated guess by the engineers as to how long it'll take to fix.
"That tells you how much of your engineering resources you can afford to dedicate to the issue. There's an upper limit as well. Unlike development, where you can develop multiple features in parallel, you can only really have one team working on a bug. If you put too many people on the bug, they won't be able to coordinate their efforts, and you'll have a lot of wasted work."
And speaking of debugging: "There also aren't a lot of people in the world that are good at debugging. The more complex the system gets, the harder it is to find someone that can figure out bugs in it. And if you can't reproduce the bug in your test environment, it's even worse. You have to work on a guess about what could have caused the varied reports of symptoms. For example, before the region crashed, all the monsters went crazy. Every time the region crashed, Bob was just about to kill the third head of the hydra with his spectral pet that was buffed from a dead player. Etc."
As obscure as these bugs can be, finding them is insanely difficult. A brand new MMO has millions of lines of code, and programmers in crunch mode aren't exactly famous for their documentation. Finding these bugs in games with a few years under their collective belts is even harder. Every new programmer to pass through the company leaves his own stamp on things. Every programmer who leaves the company takes his knowledge with him, and that includes his knowledge of how he did what and why. So, at what point is an MMO such a mass of spaghetti code that it's a miracle it runs at all?
Andrew is too modest to claim to have an answer that applied to the entire MMO industry, but he did say he was starting to think the answer relates to the number of coders that have passed through the company.
"Programmers always think most other programmer's code is ugly," he said. "Code is written in the way we think - programmers as a profession are like a crowd of self taught writers. Courses focus on the vocabulary and sentence structure, but consider style as something not worth teaching or considering. So when new programmers come in, they look at what everyone else wrote, and they decide that they don't like most of it. It doesn't make any sense to them, it's hard to read, etc. And they either isolate their work from it, or rewrite it."
One solution is management: "If you have a consistent lead programmer that's basically forcing people to write code in a certain way or leave the company, then your code at least looks consistent. That means that someone new coming in can either say "Wow, I really hate this," or "I can deal with this."
Professional management in a young industry is a challenge, of course, but it's doable. But if the company is plagued by heavy turnover among the leads (for reasons both the studio's fault and not), having a consistent lead doesn't help. Maximum effectiveness calls for the same lead to run the show.
He points out: "When you have five lead programmers in three years, it's not so clear at the end. Almost everyone hates some things - different things - and almost everyone is okay with some things."
One thing everyone is okay with is making the game stable. But making repairs isn't a matter of unplugging a widget from a socket and replacing it with a new widget. It's more like repairing smashed rear fender on a car, for a customer who just wants to get back out on the road ten minutes ago.
"[Problem solving is] also a matter of intelligently deciding when to hack a solution in, and when to redo part of the design. I used to think that all engineers really wanted to *redo* the design when structural bugs came up, and they only grudgingly backed down when they realized that the effort required is unrealistic. I've met some engineers, though, that seem to prefer to hack around everything. So the code they write tends to have hack on top of hack - which can end slowing development as much as rewriting things constantly."
I can smell the comments from the peanut gallery in response to that one. Let me attempt to forestall them by saying that an MMO studio tends to judge its employees primarily by results. If a solution to a problem with the live game is produced quickly (especially if the problem is preventing the paying customers from playing, and they're raising hell over the delay), the only question will be "does it work?" If the answer is yes, the employee gets approval. Sure, if there's time to pull out the dent, fill it in with bondo, sand, repaint, wax, and cure, any decent programmer will want to. But even the best often have to settle for yanking the metal away from the tire and spraypainting the exposed metal. It makes more word down the road, but it does get the car on that road.
More Developer Perspectives Features:
More Columns:
More Features:
Well, as a programmer, on the few programs I have written for my professors, even fewer as a gamer, doing comments on code becomes a big pain in the rear. For anyone who has taken OOP (Object-Oriented Programming) will understand the importance of comments but also understands the amount of time it takes to document that code properly. One of my professors, Dr. Thomas Turner, studies and writes compilers and debuggers and I just finished taking a class with him over Translator Design. We developed a sub-set Pascal compiler using Lex and Yacc. When he produced code for us to use as an example, he did not really have time to develop proper comments in his code and I can tell ya, that stuff is impossible to understand just looking at code.
But about the development of "Amazon" type MMOs vs. small but substainable games is a very hard issue. One thing I see is that if you make a small but substainable game but experience explosive growth. As a small developer, you might have to do what AV is doing and limit your sales to be able to accomidate that growth. But if you took the "Amazon" approach, you really do not care about the number of people who buy, you just take it all at once and deal with the problems as they come. I think trying to develop a game with a certain number of people in mind is a complex beast to handle.
In the end, I just hope future MMO companies will just try and develop core ideas and mechanics for their game. Leave the extras for patches or future expansions. Like for EQ, people will buy the expansions if they love the core of the game.
Another great read...
"The application to the MMO market, I think, is that if you want to compete now, you need to start small and sustainable. You need to build something for 1,000 or 10,000 players, and then build it better one step at a time, making money constantly, and drawing people to your product over a long period."
This part just smacks of how EVE has evolved over the 6 years.
This pretty much explains why games seem to have hit a wall in how big it can be. What used to take 3-4 people in their basement a few months, now requires 100+ for 3-4-5 years. Such complexity creates an impossible effort to keep up on the part of the debuggers.
The mention of needing to start small and build on it over time is the stance many companies seem to have taken, but there is also another solution which I think is more interesting. Instead of making trash code that gets destroyed after each game, companies should really start recycling them. Instead of writing "ugly" code with confusing comments in a rush, spend twice as long to make it so other programmers making other games can reuse it. It would greatly reduce productivity for a while but eventually you could build a game almost completly with these. Of course this could only be done by big names like Vivaldi and EA to be profitable, and probably eradicate most small developers, but I don't think games will evolve much more unless they do.
Good article -- we need more like this one.
That aligns pretty doggone closely with the coders I've worked with over the past 10 years too! The only time it didn't was when I worked for one of the Big Five (when there were still 5) and the coding had to follow a rigid structure. And just like the article says -- you either do it like that or hit the door.
The "car maintenance" analogy is disheartening but has matched my experiences in MMO's for the past 5 years!
I'd agree with this in general.
Part of the issue with larger teams is having to mesh all of them together. Programmers and Artists don't always understand each other... This leads to issues. Even within the same "type of job" you are dealing with ego, attitude and the fact people operate at different levels.
Companies used to often be very small... a group of enthusiast that wanted to create something together.
As opposed to a group of strangers being thrown together and trying to create something.
Then you also have the issue of what a Corporation is going to offer a programmer to create this code that can be recycled. Then take that same person who would actually be qualified... and take a look at what they can make somewhere else. If you are going to work for far less at a game company then you could make else where. You are either a game enthusiast who wants to create (positive) or you aren't qualified for the other job and take what you can get... which in my opinion often explains why we see the issues that the article talks about...
In general corporations don't have the mentality to actually understand "the small team" concept... let alone "its done.. when its done". If they are willing to spend the money and time to create what you suggest... then the profit comes in the long term. That is actually a good thing as its long term and sustained...
Most of the bad I see in the industry is short sighted things done for what is short term (but right now) benefit... that ends up having a very long term negative.
Hmmm, let me guess next weeks installment will be titled: Snake Bites Tail.
Episode summary: Code-monkeys defend their backside by discussing how tough it is to write solid code with so much going on with a product in development. Then the code-monkeys attack with their hairy gnarled monkey paws pointing at server-side's inability to drop code into a root directory. Code-monkeys mood lightens up and they all snicker at server-side's inability to write a server monitoring script.
Episode finishes on a positive note as code-monkeys and serverside all crash the now empty team lead meeting for leftover donuts and bagels.
Cheers All.
Excellent work! hi5
Good read. I currently work for G.E. Medical (Formerly IDX) as a network engineer. I have interviewed with a couple different development houses in my day; it usually goes like this. They show me the topology and the infrastructure, tell me what the hours are and the yearly salary…. then I laugh and walk out the door.
My hat is off to all of those stalwart individuals who do an impossible job that has crazy hours and very little pay (comparatively.) I also tend to put these guys at the very top of the interview list when we are hiring because they’ve seen fire and they’ve seen rain.
Excellent article, and a good idea for a team of programmers interested in making a litlle money.
A lot of the game engines being used are coming from FPS games. They simply don't scale well to MMO's. Vanguard did this. What some adventerous studio needs to do is develope a strong, flexible, well docomented game engine for MMO's, and then market that engine.
Excellent article. 100% agreement, Andrew clearly understands the f'ked up game dev business.
Having been there / done that for a couple decades, what I still don't understand is -
why can't we (game coders, programming dept. managerst, project mgrs, producers, etc) FORCE studios to adopt SOFTWARE ENGINEERING STANDARDS.
Documenting code should be Mandatory, weekly Peer Code Reviews should be Mandatory, conforming to studio style sheets should be Mandatory. Non-game software engineering companies (ie networking cos, database cos) do this as standard procedure - WHY NOT GAMES?!
imo managers and Producers need to become intolerant of Prima Donna ego freaks and enforce conformity to Standard Code practices.
The only reason this crap goes on year after year, project after project is because management alows it.
I'm not sure how other schools out there teach coding, but my school taught coding by the book. Meaning, there are rules you have to follow in order to get the code to work, which every coder knows. If the rules aren't followed, it doesn't compile. Now the part that doesn't make the code work any better, but takes more time to do is style. My school taught the by the book way on how to write code, so that anyone picking up the code would understand it. That includes the notes and everything that goes in the code to let programmers know what they're looking at.
So reading about excuses on how code isn't recognizable from one programmer to the next is inexcusable, considering that in my school our students were taught the formal way how to code for that very reason. So in my opinion, either the coders didn't pay attention to their classes, didn't care, or wasn't taught. Either way, the point being is that the company hired coders who may have helped create a game, but they're causing everyone to work harder due to their lack of professionalism.
Maybe the IT profession should adopt a standard style of coding, so that no matter who picks up the code, they can read it. Sort of like what the accountant profession has done with their GAAP.
I would like to hear more on how often they test their code?on average that is.
I can see the importance of comments for finding problems later but that would take a huge amount of time to add in that many over millions of lines of code,it would add a lot of time to production.I would think if they actually test their code often enough ,they really should not have any or many problems.
I have seen some pretty lame bugs in games,that NEVER get fixed,and seem quite easy to fix,so they must have very low standards that a bug has to be incredibly game stopping before they fix it.I hate to say it but SOE is kind of the worst for fixing bugs,they will allow MANY to stay in game if they do not stop or make game altering changes.
I can give one example from EQ2.There is a spot in one of the zones,it is an older zone the forest type zone,with the water way that runs from top to bottom,i cannot remember the name off hand.In this zone ,a tad before you climb the stairs,if you stand in one spot,the entire graphics/map go blank,there is no visual.Sure all players can continue their game,they won't get stuck,but it looks lame,and should have been fixed ,but not after what?5 years ,still not fixed.IMO it is a mapping flaw,and the guy who designed it i would think would feel embarrassed by such a poor mistake,that he would want to fix it pronto,guess not.
Once again in EQ2,certain mobs would spawn with their mesh all clustered and messed up,the head might be on the bottom and the arms and legs are scattered in different positions,they never fixed them especially the skeletons in one of the main starting zones,again i find it extremely lame to leave something like that for many years untouched.How about mobs spawning underground,so players are sending in massive complaints,they can't find a certain boss,all this seems like it should have been noticed in the simplest of testing[before the game was released],so the amount of debugging that goes on for many developers must be low end or not as often as it should be.I guess it is the testing that seems to be lacking for most developers.
A bug i have seen often in MOST games,is one where mobs can often attack you even when you are suppose to be out of reach,i have never seen one of these fixed in any of the games i have played,so again developers do not seem to care much that their product has bugs unless they are game stopping,they must have no pride in their work are are afraid to spend any resources on the fix.
An example would be when climbing a ladder,the mob is still on the ground and you are miles above on the ladder,yet the mob is still whacking away on your hit points..lmao lame.On a whim i can only guess they did not make an animation of the mob climbing the ladder,that is why he is probably up the ladder with you but there is no animation so he looks like he is still on the ground,IDK just a guess.This is another simple example that would have been noticed with the cheapest of testing,so again someone in charge of the development side is lacking in their efforts or afraid of cost to test their product.
IMO many of these games are biting off more than they can handle,if they cannot afford a steady test/debug team,they should not be making the game,otherwise they are laughing in the buyers face,figuring we will buy any junk they throw at us.I guess this is true from the many forum posts of players claiming a game is god beofre it even comes out and the many buying pre orders..pffft.
That was a great article. As a programmer in the business world, it was interesting to read Andrew's comments about programming... also interesting to see that the same problems I see in the business world also exist in the
real worldgaming industry.Lol, like half of the problems with games wouldnt exist if programmers had any real-world programming skills. Yeah modern games are a very complex piece of software, but so is every CAD system, OS, even Office. They all have bugs, but 90% of the time its simply programmers fault. Some random genuis decides that a function doesnt need to validate/restrict its input parameters and you get a vulnerability. Or he's just too l33t to actually test the code. Programming is based on simple rules, if you stick to them your software will work as a charm. Its not like the program has a mind on its own and decides to act weird all of a sudden, but then again its easier to just say 'its so damn complicated there will always be problems with it'.
I would like to see Andrew commenting on things like Mythic seconds for example and all those hundreds of small bugs like weapon claiming its actually a hat :P
Oh forgot something, how many times have we seen changes missing from the patch notes? And im not talking about changes that are supposedly targeted at exploits and such. Its anoder 'standart' game programmers should get rid of
Lets compare a MMO with a car.
I don't care how hard it is for the mechanics to fix it I just want it fixed as soon as possible and I don't want to listen to their excuses why it's taking so long to have it fixed.
The application to the MMO market, I think, is that if you want to compete now, you need to start small and sustainable. You need to build something for 1,000 or 10,000 players, and then build it better one step at a time, making money constantly, and drawing people to your product over a long period."
Darkfall anyone?
Because time (budget) is limited. If you only get the time to do things quick and dirty, then thats all you can do. Doing everything with "normal" software engineering standards would crush 75% of all projects out there. The current methods just use up too much time, no matter how "good" they seem to be. Every programmer and designer has the knowledge about procedures, models, processes etc. but in most cases its either do it fast and in time or not at all. So choose: do you want 10 projects with enough budget for a full blown development system or 100 projects that do it quick and dirty?
Sounds like Andrew has worked with a few coders down the hall from me. I have to commend those guys on their jobs, in that aside from sometimes slightly better pay than us artists, they get no recognition for their feats of engineering genius. The designers and directors put impossible odds on them. Artists at least have our own little communities where we might share models, rigs, or keyed animations, that help us cut corners when we need to meet a deadline. Not the coders, unless they wrote the lines themselves, they often have no idea what a section of lines are actually supposed to read like. Add to that constantly changing minds in the higher ups, and you have impossible amounts of lines that should have been recoded, and got glossed over, or a type-o hidden in the code. Sometimes looking at the flow charts for our animations gives even me vertigo, so I dare not imagine what it would be like to have to read millions of lines of code and double check every one of them (what is being asked of them, but never the time to deliver), especially when they have been staring at the same lines for the past 3 years, for sometimes 80+ hrs/wk.
It seems strange but when i was at university i was very good at problem solving, yes generally my own coding wasnt that great but i could problem solve other peoples code very quickly.
The problem is now im very much out of touch with my programming i stil have the skills to work around problems as im doing it every day in work with the really bad Excel spreadsheets companies insist on using instead of proper data bases.
I even rewrote one of the works spreadsheets so it would give constant updated summarys of what i need to do as well as the banking one it gives the team leader a complete breakdown and running total of how much money we have managed to claim back (its sort of debt recovery).
It was always one of those aspects of coding that i never understood, "documentation" until when i was at university i had to debug LOADs of friends code. It would be a cut and pate nightmare and very few seemed to understand basic coding principles. Even pseudocode would not be clear. When i have a problem i normally write down how i want to solve the problem i then look at my tools and see if its feasable.
Still it would seem strange that the engineers working ON mmo's wouldnt have a dedicated team set asside who would work on the code that after a hack goes in then reengineer that hack into the core code to make it more efficient. But then i am a perfectionist at heart and doing somthing half arsed seems to be morally wrong. If a jobs worth doing its worth doing properly.
But i suppose time = money and thats the motivating factor in a lot of jobs.
Very interesting article: I espeacially liked the end parts about the philosphy of starting a new MMO. Definitely explains why so many niche games are popping up but no major contenders stepping into the ring. To all the developers making small sustainable games, I say to you have you ever heard, "No guts, No glory!"? Personally I think its difficult to consider any MMO a sucess that can't attract a large diverse audience. Economcally MMO's tend to be liek movie grosses IMO....they start large and trail off as time goes by. The industry as a whole now seems to be opting for straight to video than hollywood blockbuster. Adding features gradually in hopes of finding your player base is just as big a gamble as blowing your wad on one big grand idea....it only affords you more time to access your sucess or failure. But if anyone is gonna break the WoW barrier ( i guess thats liek the sound barrier for games) some developer has got to have the cajones and the infrastructure in place to put a grand idea out there. I think KOTOR's got a good shot of being that game but maybe still too early to say that. Anyway, great article, very interesting reads.
tweety?
the same tweety from lum the mad and her worthless rants from back in the day?
yeah i'll skip this blog .
more bout her sorry ass here
http://www.geocities.com/thetruthaboutlumthemad/
Great read sweetheart. You really did a good job with this one, always wondered about the underbelly, not yours, already saw that one twice - penetrated.
anyways, what does this mean for vanguard, does this mean vanguard is the worst made game in history, because they have gone through more programmers in one year than wow has since it's been out.
Vanguard? Why do you even mention it? I think with this "talk" Andrew shot Mythic in the head and buried it. Come on, the guy is the Technical Director there in MYTHIC not anywhere else. Do you really think he is talking about anything but Mythic and Warhammer? Andrew explained why WAR sucks – simply and clearly. I personally thankful for that but Mark, Paul and so on have to be really mad.
tweety?
the same tweety from lum the mad and her worthless rants from back in the day?
yeah i'll skip this blog .
more bout her sorry ass here
http://www.geocities.com/thetruthaboutlumthemad/
Wow…just wow. Folks tell me not to clink the links but you know me; it’s shiney new link I just had to click.
First I’m amazed that Geocities is still around (Hello 1994 how are you?) But then I see this losers rant about crap that happened (his screwed up jealous version of it
I was one of the original Lum posters and before you label me a Mythic fanboi you might want to consider that I have more disciplinary action from that site than Lindsey Lohan has fake prescription pads. Idiots like you whined continuously that Lum sold out no matter what he did…the guy couldn’t win. And the only thing Tweety is guilty of in all of this is not smacking your sorry asses down hard in one of her op-eds. Now here is something that your shrink should have told you long ago….let it go Climb down from your cross use the wood to build a bridge and get the hell over it.
We are sorry you didn’t become a person of note on the interwebsz It isn’t really that fun. I have seen the crap two of my friends who have become household names in MMO gaming go through every day…being libeled by twits like yourself who doesn’t even know who they are or what they are about. The sad thing about all of this is any waste of skin can put up a site and do a smear of anyone else. Well you have joined the ranks of thousands of other no lifes…and brought attention to your sad little world…aren’t you so very proud?
Wow…just wow. Folks tell me not to clink the links but you know me; it’s shiney new link I just had to click.
First I’m amazed that Geocities is still around (Hello 1994 how are you?) But then I see this losers rant about crap that happened (his screwed up jealous version of it
I was one of the original Lum posters and before you label me a Mythic fanboi you might want to consider that I have more disciplinary action from that site than Lindsey Lohan has fake prescription pads. Idiots like you whined continuously that Lum sold out no matter what he did…the guy couldn’t win. And the only thing Tweety is guilty of in all of this is not smacking your sorry asses down hard in one of her op-eds. Now here is something that your shrink should have told you long ago….let it go Climb down from your cross use the wood to build a bridge and get the hell over it.
We are sorry you didn’t become a person of note on the interwebsz It isn’t really that fun. I have seen the crap two of my friends who have become household names in MMO gaming go through every day…being libeled by twits like yourself who doesn’t even know who they are or what they are about. The sad thing about all of this is any waste of skin can put up a site and do a smear of anyone else. Well you have joined the ranks of thousands of other no lifes…and brought attention to your sad little world…aren’t you so very proud?
Wow, now I'm interested! Seems like fun!
I really brought that on myself, by clicking on that link. I have no sympathy at all, for me! I thought I would see a flame-storm that'd make the Darkfall threads look like Barrens Chat. Instead, it was just... whiny. How sad. People get way too attached to the forums they frequent, and I don't know why.
Naw, you're wrong, I read it and it wasn't whiny, lum has a point, and i won't ever read another post by tweeter.