Kole pressed his hand to his forehead again, and wiped the sweat from his brow. His headache was worsening, and he felt uncomfortably hot, even though his body shivered. Not now. Not a fever, Kole thought. I gotta get back to work. He used his coat to wipe the perspiration from his fingers and grabbed his keys.
The car clock said 5:40am. He’d left work three hours earlier to grab some food and rest, but he couldn’t sleep, not with all the issues at work, not on launch week. A few years ago he’d been hired as a game designer for Deluvian Games, and this week was launch week, or what employees referred to as Day One.
At the coffee shop, Kole couldn’t help but eavesdrop when he heard his company’s name at a corner table.
A guy in khakis was talking to two others. “Since we don't actually know what went wrong, or at least I don't, we have two questions to answer. “How did Deluvian test their infrastructure? And what infrastructure did they use?”
“Do they even care about us players?” Another of the three asked.
Waves of guilt washed over Kole, making his fever feel like an afterthought. It had been just under two years since Kole was added to the project, and for the last twenty-two months his whole focus was to get this game ready for launch. But they’d run out of time. They tried, but failed. They’d put in over two thousand hours of overtime, but it wasn’t enough. Launch day was here and there were issues—server issues. His first major online game launch and his only thoughts were, Could I have done better? and, What the hell can I do to fix it? Kole sipped his coffee, stole a final glance at the table, and left.
As he walked into the madness that filled the company’s halls, dozens of employees lined the entrance, almost running into him. Some muttered into cell phones while others directed frantic duties. Kole turned down the hall that led to his desk. He noticed all of his neighbors were either scribbling notes or pecking furiously at their keyboards.
When Kole reached his office, Jon, a lead engineer, was waiting for him while shouting orders into his phone. “Well, there are three primary ways a company can host their games: own all the servers, rent all the servers, or use the Cloud. We own all ours, so get someone who can get down there and get it done!” Jon hung up the phone, and before Kole could offer a greeting Jon motioned for him to follow. “Meeting in the Bull Pen. Now.”
)
The meeting room was a wreck, much like everywhere else. Worry masked every expression.
Jon took the floor. “Okay everyone, listen up. Our forecast for Day One was wrong. No excuse. We don’t have the funds for additional infrastructure, so we have no choice but to wait it out instead.” Groans filled the room. “We’re starting rolling queues, and we’ll have tiered access to ensure a more even load. It is inexcusable not to expect massive load first week, and you can bet a lot of our asses are getting chewed for this. It happens to the best and worst games.”
“Can we get more servers?” Someone asked.
“I want to buy more hardware, but I’m running into a brick wall trying to get it approved, because we can’t expect the first week player load to be our regular run rate.”
Another spoke up. “But we should have expected it to be well above our normal operational forecast, and should have been prepared for it.”
“I agree. And we’re trying to get it fixed. We own and operate all of our servers. One of the things we can offer other developers in a publishing deal is the ability to host. This also makes all the sense in the world as to why we were unprepared for the Day One spike. It’s just too expensive. Day One concurrency can represent a larger than normal amount of load. To invest in hardware we would only need for a few weeks would be $10-$13 million, and that’s if we had the room for the extra hardware in our facility which we don’t. And we don’t have that kind of revenue.”
An employee next to Jon started to speak. “One of the problems is the number of API calls made to the server created a significant load. This problem is common in MMO's, and a huge amount of time is spent streamlining the API budget to make the game lag less, and cost less to operate. The later this process is started the bigger a problem it is. What we should have done and are trying to do is load balance between our servers, and host servers who can take up the extra load during Day One. No excuse for not doing this other than wanting to only use our own hardware.”
“Once again, this is going to cost a lot of money, but we’re going to do everything in our power to try and help our players get online.”
Another employee jumped in. “We should have tested more thoroughly.”
Then another. “We had the same problems during beta, which means we knew things were going to break.”
“We ran out of time.”
“Somewhere, in every game company, there is a guy with a calculator and a spreadsheet that says the game has to release by this date. Ready or not, we had to release.”
“We should have managed the server load, balance the load between internal servers, and hosted servers for the Day One spike.”
“Yeah, if only—”
“It’s not that easy—”
“Where was engineering?”
“Who’s IT talking to—”
The room erupted in a flurry of shouts.
“How much did QA see with the beta?”
“Why wouldn’t the producers give us more money?”
“My department didn’t see anything until June.”
“That was the quickest we could ship!”
“Wasn’t quick enough—”
“But what about our players?” Kole shouted.
“We need this fixed now, so they can get online!”
“Players matter. Nothing else comes close.”
“We let them down.”
“We have to make it up to them.”
“Your department has the most making up to do.”
“So does yours!”
“I thought you’d be the first to know a little more than nothing!”
The room continued its uproar.
A meager looking fellow moved to the center of the room and motioned to Jon that he wanted to speak.
Jon stood on his chair and yelled. “That’s enough! Listen up!”
The small fellow pulled his glasses off his nose and glanced around the room. He cleared his throat. “I am the guy who forecasts game launches for things like this—”
The room seemed to collectively gasp, and all attention suddenly shifted.
“See… I… when I forecast…” He cleared his throat again, and kept his gaze toward the floor. “I forecast how much server capacity we need. I look at other games, look at the beta signups, look at pre-signups and so on, but it is not an exact science, so it is very difficult. Folks in my position are taught that Day One is going to be 300%-500% over normal run rate, and so normally you plan for that. You just do it, because it happens every time and if you don't, you weren't paying attention. I thought I was paying close enough attention, but obviously I was wrong. I cannot begin to express how sorry I am. Don’t blame each other, don’t blame yourselves, blame me. I’ve failed every single one of you, and every single one of our players. I can’t make it up to you. I can’t even begin to make it up to our players. But I’m going to try anyway. Listening to all of you just gave me an idea on how we can get everyone online this week. If we all work together, I was hoping—”
“—anything you need.” Kole blurted.
“Yeah. I’m in.”
“Me too.”
A new tone began to fill the room—congenial and encouraged.
Jon stepped forward again. “We’ve all made mistakes; we’re human. We can’t blame each other, and expect it to resolve anything, but we can work together, harder than ever, and make this right for our players. So, let’s go!”
)
This short work of fiction was inspired by the conversations we had on the Game On show hosted by Adam “Ferrel” Trzonkowski and Chris “Syeric” Coke. We talked about MMO launches and how and why they might have hiccups. I whipped up this quick short story based on various developer testimonials to try and illustrate how our industry partners in development might feel on launch day. I hope you enjoyed it! To listen to our full discussion, visit Game On: FFXIV & Botched Launches.
P.S. Find me at GeneseDavis.com and on Facebook and Twitter.
Every week, Holder’s Dominion author Genese Davis opines about MMO gaming, the issues the genre faces, and the power of shaping online worlds.
Check out more columns by Genese:
