Dark or Light

Timesink II: Revenge of the Timesink

Shannon Drake Posted:
General Articles 0

Alright, the game shipped, now everything is okay, right?

Everything is alright for a few minutes, perhaps a day if the launch is going well--it usually isn't, just as a rule, even a sane launch on the outside can be a bit of a mad house on the inside--and then it's time to start working on all the things we've neglected and finish polishing up the game. That said, at the end of any launch cycle, there are a number of confounding variables that may throw a monkey wrench in the metaphorically smoothly operating bug-fixing machine.

One of those monkey wrenches may be everyone quitting. At the conclusion of any project, there's a wave of developers heading for greener pastures. Some quit the industry entirely, sick of taking a pay cut and working ridiculous hours for having a cool title. Some just want to see family they haven't seen during the 6 month to 3-year launch deathmarch. Some are sick of being around the people they've been around for 8-24 hours a day for the past 2-5 years. Some want to start their own company. And some just want to do something, anything else. There's another factor at play, too, which is "games shipped" get you higher levels of industry cred than "games successfully maintained." Of course, this wave of departures is a significant brain drain, as the people who spent years getting to know the game while they built it just left.

That's not accounting for the other classic gaming industry maneuver, which is, "Thank you for all your hard work and for sleeping under your desk for the last year or so, now get out." Even modestly successful projects may have a wave of layoffs at the end, which not only results in everyone who gets laid off leaving--and a guy stuck looking for his next job isn't exactly going to be eagerly documenting all the secrets of the engine for his successor--it also results in a further wave of departures as people spooked by the wave of layoffs depart for greener pastures, as very few people are eager to stay on what may be or may seem to be a sinking ship.

This wave of post-launch layoffs may stem from a genuine need--sales say we only need to support 100,000 people, not the 1 million we staffed for--or because someone in management realizes the savvy crew of 3-5 year veterans that just launched the game cost a hell of a lot more in pay and benefits than a bunch of fresh out of college kids willing to eat Ramen and live out of a box to work in ~*~The Exciting World Of THE GAMING INDUSTRY~*~, even if the college kids have no real idea how the game functions. That's assuming the team actually got to launch before the axe swung, as there are a bevy of exciting ways to be laid off in the industry. No matter what the case is, the loss of the talent that made the game can be a huge obstacle to successfully fixing broken things post launch, especially in the more-arcane systems that only one or two people may understand.

Be it quitting or layoffs or other reasons, there's, there's usually a smaller crew working on the game after launch, usually with some fresh blood that may not know everything the old crew knew. And that's before we get into specific reasons why things may not be fixed up quickly.

Those may include:

We're All Gonna Die!

If there are a lot of critical bugs that crash the server or otherwise make the game unplayable, the team may spend the first few months firefighting on critical issues and anything that doesn't induce arm-flailing panic gets back-burnered until all the bad stuff is fixed. Depending on the luck and abilities of the team, this could be a matter of a few patches or it could be a matter of weeks or months to get everything fixed and working, and then it's time to tackle all the exciting new bugs.

It Shipped Too Early

Not that anyone is eager to shove a half-complete game out the door, but time, publishers, and men in suits may insist it actually has to go out even if everyone in the world knows it's a bad idea. Launch is not an easy thing to push back, as there's a great heaving mass of machinery behind any major launch. Pushing back requires things like rescheduling print runs of boxes, rescheduling burning the discs, changing all the plans with the retailers, changing ad buys and ad copy, resetting press schedules and events, renegotiating vendor contracts, and otherwise rebooting everything imaginable. At some point, launch train haven't brakes.

Given the spittle-flecked quivering rage of the MMO masses at a feature promised and denied, some attention may go to features that didn't quite hit the launch window. Depending on how important they are versus the exciting new bugs discovered on launch, they may be prioritized accordingly, which means the team may spend months playing catch-up and getting the things on the box actually in the game, rather than working on all the broken-but-not-critical stuff.

Content For...The Future!

It's no longer enough to ship the game and maintain it. More and more developers and publishers are committing to a whole slate of regular non-expansion content, which, of course, requires developers to work on it, test it, and get it out the door. That said, the future content may be introducing new and exciting bugs on top of the already existing bugs and if the emphasis is on pushing the new shiny thing out the door to sate the hungering masses, the little annoying things might just have to wait. This depends on the size of the remaining team, of course, some companies have a dedicated team for fixing old stuff and a separate dedicated team for making new stuff, though if the axe came down after launch, these both may actually be the same team.

Planning for...THE FUTURE!

Given the long development time of one of these beasts, it's not uncommon for expansion planning to begin in earnest months or years before the original game ships, especially if the initial sales figures are strong. If it's a multi-game shop, wannabe or otherwise, the team that's just finished launching the game will probably start shifting to the next game to begin laying the groundwork there, leaving a smaller team to work on the existing game. Given the relative level of excitement between preparing a new product--exciting!--and maintaining an existing one--kind of boring!--there may be a herd of developers racing to volunteer to be on the new team, rather than maintain the existing game.

That's all before we get to the actual bugs.


The specific system varies by company, but there are roughly five tiers of bugs in any QA system. These are roughly described as:


Serious, critical bugs that render the game unplayable for large masses of people like server crashes, major bugs in the client, the bad things that keep the game from running for everyone. These usually get fixed as fast as possible, though if it's something particularly tricky, it won't be fast enough for the public.

Bugs Are Serious Business

Bugs here are important enough that they need to get fixed, but they aren't catastrophic. These will get fixed once the red alert emergency bugs are fixed.

Eventually Fixable

Stuff that needs to get fixed one day and probably will, but isn't as important as the previous two tiers.

Annoying The Hell Out Of You Personally

These bugs are the tiny little things that will get fixed when a coder gets bored and needs a break. This is inevitably the one little thing they could just fix in five minutes that is annoying you, personally.

Probably Never Going To Get Fixed But We Can't Exactly Say That

There are certain things that are so arcane, difficult, or convoluted that they will never get fixed. There are legends in most large software applications of a section written by a mad genius way back at the beginning and never quite documented. Its use is critical to the application, but no one is quite sure what it does and the few times it HAS been messed with, bad things have happened, so it's best to leave it alone. Some things are small enough that they can be safely put off, though if there are new bugs and regular releases, they may be put off...and put off...and put off. Some are only affecting a few people. And sometimes, the systems and mechanics of the game are arcane enough that tweaking them seems minor, but would require making a major effort to actually fix.

So, why is fixing things so hard?

For one thing, you're dealing with tens or hundreds of thousands of lines of complex code written by 20-100 different people and given the increasing globalization of every industry, it wouldn't be uncommon for significant chunks of those coders to be in places like China, India, Eastern Europe, or other outsourcing hubs. If the game uses third-party software, add in lines of code made by a different company with different coders, standards, and ways of doing business.

For another thing, PC games run on a variety of different hardware made all over the world by lots of different companies. A console developer has to account for 1-3 different styles of console that are usually variation of the same hardware. A PC developer has to account for 2 major video card chipset manufacturers and whatever minor ones they care to support, 2-3 major operating systems if they're still supporting XP, 2-3 major chip manufacturers, and a delightful array of motherboards, sound cards, network cards, and various other hardware, on top of which is an ungodly combination of software ranging from staples like Word and instant messenger to "granny downloaded a screensaver promising her cute pictures of kitties and now her computer is running slow." Supporting certain hardware configurations is the usual route to go, but makes the assumption that people actually read labels and warnings, but they don't. If you were to sell KILLZ U DEAD with a bright red warning label saying DO NOT DRINK THIS IT KILLS YOU DEAD, you'd still get complaints from dying people claiming to know the president of the company, demanding a refund, and rghrrhghrhghr...

And then there's the time issue, which usually comes down to "Would you rather we fix some annoying things, work on cool things, or systematically dismantle the engine to solve that one little annoying thing that's bothering a few people?" Things that seem simple inevitably aren't simple and given the complex nature of any project, tinkering with one system can lead to unintended consequences throughout the system.

Assuming a bug is worth tackling, it requires writing and editing the code, compiling it and building it, testing the patch on a variety of machines, making sure it's actually working and fixed what it was supposed to fix without breaking anything else, getting all the signoffs for everyone involved to make sure it's actually working, scheduling a push to the test server if there is such a thing, putting it on the test server, making sure nothing goes wrong with it THERE, scheduling it to go out on the live servers, taking all those servers down, putting out the patch, bringing the live servers back up, testing the live servers to make sure something hasn't screwed them up, opening the servers back up, and monitoring for anything not caught in the previous processes. All that is assuming the whole process went smoothly.

Anything going wrong in any one of those steps can send it all the way back to the beginning, and a team that tries to take shortcuts will inevitably discover why all those processes are in place, especially if they have bad luck, and especially if it's the wrong day of the week to get the patch out (Thursday and Friday tend to be especially bad days as people leave early or take off for the weekend, go out of town, disappear into the ether, go have what they can of a social life, etc.). That's not to say everyone can't come together in a crisis and things can't go just perfectly, but thrashing around in a constant crisis mode tends to burn people out and lead to the brain drain at the beginning of the article, and doing things quickly tends to result in carelessness that leads to more problems.

That's assuming the numbers say problems are worth fixing. Games that tank might be left in maintenance mode with a small team dedicated to fixing the worst problems and the rest of the team either laid off or working on the new project that will make up for all the money spent on the tanking project. However, what they lack in manpower, they often make up for in enthusiasm, and the team on Project Red-Headed Stepchild may hurl themselves into the task of fixing their game up, but their resources are necessarily limited by the size of the project and whatever resources they can scavenge.


Shannon Drake