Trending Games | Elder Scrolls Online | WildStar | Guild Wars 2 | TERA: Rising

  Network:  FPSguru RTSguru
Login:  Password:   Remember?  
Show Quick Gamelist Jump to Random Game
Members:2,645,814 Users Online:0
Games:687  Posts:6,084,421

Show Blog

Link to this blogs RSS feed

Jumpgate Evolution Developer Blog

Jumpgate Evolution is a Massively Multiplayer Online Game set in the open expanse of space. With breath-taking visuals and an innovative twitch-based space combat, Jumpgate is the definitive space combat MMO, putting you at the heart of the action.

Author: JumpgateEvo

Contributor: Khatie

Build and Deploy - Hermann Peterscheck, Producer

Posted by JumpgateEvo Tuesday October 27 2009 at 2:51PM
Login or Register to rate this blog post!

After much deliberation I decided to devote this blog to one of the unsung heroes of MMOs, which is the developers that are in-charge of building and deployment. Now before you all check-out over what you think sounds like a boring topic, I want to remind everyone that without a decent build and deployment system, no one will be able to play our game.

As you guys have heard many times by now, Jumpgate is built on the concept that iteration is a central requirement for quality. This begins with the build and deployment process. I’ve worked on lots of software projects where getting a build together was a nightmare… getting it out the door? Ha! Almost impossible.

Here’s how things normally work on such projects. Everyone is kind of working on their own stuff, on their own machine; chugging along, making awesome progress. When they feel that it’s at a functioning state they check it into whatever version control software they are using. At some point (usually every 2 or 4 weeks) “build day” comes along. It is at this time that all these wonderful changes everyone made come together and…. BOOM the whole thing explodes. This is almost always on a Friday, or even better, just before a holiday, which results in people spending late nights and/or weekends trying to remember everything they did for the last 2-4 weeks that might have caused the explosion. It’s lots of fun!

On JGE we set out to avoid this kind of thing. From the beginning of the project we setup a system by which all changes are made in a central location. Once a change is detected our build machine automatically makes a build of the game and sends out all the change notes to all the developers. The next time someone runs the patcher (yes our internal process uses the same patcher that we distribute), they are seeing those changes, in most cases minutes after they were made. This means we have more frequent, easier to solve, smaller explosions along the way and, eventually, very few explosions. It also means that we have to keep our build times down to about 10-15 minutes. I’ve heard of MMOs that take an entire day to build. Imagine how fun that weekend build fix is going to be. The net gain of this environment is that we have direct exposure to changes as they happen, in real time. We experience the same stuff players will experience as early as humanly possible. It also means that weird crashes and strange one off bugs are more likely to be caught early and the person who added them can fix them when the possible cause of the problem is fresh in their mind.

As the project grows you have to do things to keep a separate version of the game in good working order. So we have a separate version that artists and designers can use to keep working in the event that some developers checks something in and breaks things for a while. However, we still promote every night to keep things nice and synched as well as allowing a “programming” branch to become the artist/designer branch on a whim.

At deployment time, this is incredibly valuable. The system is mostly automated and so deploying a new version of the game requires a minimum of human intervention. By doing this quickly, it means it gets done a lot. By doing it a lot it means that we have a bigger chance of catching all the nasty bugs that make deployment messy and nightmarish. In short, by the time the game launches we have run the patcher, deployed a build, updated the database, upgraded the servers and created the client files literally thousands of times. Even with this process in place we STILL find problems with some frequency. This ties directly back to the premise of iterate to quality. See, I told you it was relevant. :)

MMOs are hideous beasts with complex moving parts that change all the time. It’s kind of like riding a unicycle blindfolded while juggling chainsaws and firing a crossbow at a randomly spinning target.

So how does any of this help you, the player? Well the idea is that by having these processes for development in place we can deliver things to you much more quickly and with greater success. Reliability is not something that MMOs are known for. Not in feature or content deployment, not in scalability, and certainly not in game balance. While some of those problems have difficult solutions we thought we could at least take some of the known risk out of the process and increase the chances that players get something they can have confidence in.

As we go into more public beta testing, the reliability of this kind of system gets pushed to the limit. In public testing is when many of the major problems with MMOs first become exposed. No matter how much unit testing or bot testing you do, there’s something about having a few thousand real people trying all kinds of things that exposes new problems. Part of what we will accomplish with expanding public testing is making sure that the deployment system is tested to ensure a quality experience for customers after a commercial release. This is a result of both frequent and rapid deployment processes.

The other thing I wanted to do with this was to give some credit to the thousands of people in the MMO industry that design, implement and support deployment and build systems. Their jobs are tough. They usually involve working late nights (after builds happen and break), deal with many organizations (development, operations, customer service and billing) and suffer front line exposure when things explode (Oh noes11! The server went down again!!!!). Being the person who sends the “new build is up” email is not fun when the new build causes massive data corruption.

So next time you are playing an MMO that pushes some new content or feature, has a hotfix or deals with some data integrity problems, give a silent thank you to the people in charge of these systems… they live in the reality that when nothing happens they have done a good job and they only hear about their job when disaster strikes.

dodsfall writes:

Testing changes locally with the latest "get" will also head off a lot of issues and broken builds.

Tue Oct 27 2009 3:49PM Report
Demz2 writes:

thats all very well and good, but what the hell is going on with this game?  Is it ever coming out?  when are we ever going to get any actual solid news on it?  Its been a dman long time.

Tue Oct 27 2009 4:29PM Report
MarineBoy writes:

 For some time now Jumpgate posts articles that are totally irrelevant to the consumer. The internal processes that are used by the company are of no interest and of no concern to us. We will however be able to tell without a doubt what a good (or bad) job you did as soon as we can get our hands on the product itself: THE GAME!

So please do us a favor, stop talking (rubbish) and keep working, maybe this way the game will be out by Q4 2010!

Tue Oct 27 2009 6:21PM Report
Lorgarn writes:

Pretty interesting interview, i love to hear how different companies manage the development progress and their building- and deployment-systems. It often varies alot more than you could imagine.

Tue Oct 27 2009 7:06PM Report
crainey92 writes:

 Somehow i managed to have great interest in this article and now feel like a complete geek. I even decided to tell my friend on ventrilo about it and i completely bewildered him. 

 

When im playing games i like to look at the designers point of view whilst playing through the content so this gives me a better insight into the gaming industry enviroment, therefore i thank you for this article. :)

Tue Oct 27 2009 7:58PM Report
orionite writes:

Thankfully, not everyone is as narrow-minded as MarineBoy. I for one enjoy reading articles like that. I can get the usual consumer targeted marketing drivel anywhere and all day long.

If you're not interested, don't read it. Trust me! It works!

Wed Oct 28 2009 12:02AM Report
freejackmack writes:

ND continues to build up my confidence in them. I had read that it took a good bit of work to get the server configuration working the way they wanted and I wanted to know more. I like to be reassured with details about how the game I am waiting for will provide a stable experience. Well done!

Wed Oct 28 2009 1:03AM Report
2muchfun0 writes:

As a well experienced MMo player I can tolerate the wait for quality rather than play a month and the game bellies up from lack of deployment testing as some past mmo's I had bad expeirences with,my thanks to ND for the continueing effort to give us what we want. and thanks for this update.

Wed Oct 28 2009 8:19PM Report
Dvalon writes:

to Quote a Legend " Need more Input!! " the amount of data on this game is a joke considering it had a stalled beta start and launch, there should be far more information on it.

Thu Oct 29 2009 6:04AM Report
GoBBels writes:

Thank you for the article Herman, very interesting.

"stalled beta start and launch"? They're in Friends and family testing, no beta(or release) has happened yet.

Thu Oct 29 2009 8:34AM Report
Daedalus732 writes:

"Thankfully, not everyone is as narrow-minded as MarineBoy."

 

There was nothing narrow-minded about MarineBoy's post. He's absolutely right. Consumers don't give a shit about the internal processes of the MMO companies. They just want to play the game.

As a consumer, I don't care how hard it is to make an MMO. As a voter, I don't care how hard it is to be President of the United States. As a patient, I don't care how hard it is to stitch up a wound or perform a root canal. All I care about is the end product. The methods might be interesting to some small group of people, but they are totally irrelevant to most people. All we care about at the end of the day is the quality of the end product, and yes, sometimes the amount of time it takes to deliver that end product effects the decision to purchase or support.

A developer blog for an MMO should talk about the game. Save the internal mechanics discussion for after people are wowed by your product. There is just so little information out right now that a blog post like this looks more like filler material than anything else.

Give us a beta date. Give us a release date. Spell out the game's features. If you can't do any of those, then just be quiet.

Sat Oct 31 2009 8:52PM Report
wikraaben writes:

Here’s how things normally work on such projects. Everyone is kind of working on their own stuff, on their own machine; chugging along, making awesome progress. When they feel that it’s at a functioning state they check it into whatever version control software they are using. At some point (usually every 2 or 4 weeks [url=http://www.watchcorrect.com/watches-by-brand/michele.html]michele watches[/url] ) “build day” comes along. It is at this time that all these wonderful changes everyone made come together and…. BOOM the whole thing explodes. This is almost always on a Friday, or even better, just before a holiday, which results in people spending late nights and/or weekends trying to remember everything they did for the last 2-4 weeks that might have caused the explosion. It’s lots of fun!

Mon Nov 16 2009 1:01AM Report

MMORPG.com writes:
Login or Register to post a comment

Special Offers