Jumpgate Evolution Developer Blog

Jumpgate Evolution is a Massively Multiplayer Online Game set in the open expanse of space. With breath-taking visuals and an innovative twitch-based space combat, Jumpgate is the definitive space combat MMO, putting you at the heart of the action.

Author: JumpgateEvo

Contributor: Khatie

After much deliberation I decided to devote this blog to one of the unsung heroes of MMOs, which is the developers that are in-charge of building and deployment. Now before you all check-out over what you think sounds like a boring topic, I want to remind everyone that without a decent build and deployment system, no one will be able to play our game.

As you guys have heard many times by now, Jumpgate is built on the concept that iteration is a central requirement for quality. This begins with the build and deployment process. I’ve worked on lots of software projects where getting a build together was a nightmare… getting it out the door? Ha! Almost impossible.

Here’s how things normally work on such projects. Everyone is kind of working on their own stuff, on their own machine; chugging along, making awesome progress. When they feel that it’s at a functioning state they check it into whatever version control software they are using. At some point (usually every 2 or 4 weeks) “build day” comes along. It is at this time that all these wonderful changes everyone made come together and…. BOOM the whole thing explodes. This is almost always on a Friday, or even better, just before a holiday, which results in people spending late nights and/or weekends trying to remember everything they did for the last 2-4 weeks that might have caused the explosion. It’s lots of fun!

On JGE we set out to avoid this kind of thing. From the beginning of the project we setup a system by which all changes are made in a central location. Once a change is detected our build machine automatically makes a build of the game and sends out all the change notes to all the developers. The next time someone runs the patcher (yes our internal process uses the same patcher that we distribute), they are seeing those changes, in most cases minutes after they were made. This means we have more frequent, easier to solve, smaller explosions along the way and, eventually, very few explosions. It also means that we have to keep our build times down to about 10-15 minutes. I’ve heard of MMOs that take an entire day to build. Imagine how fun that weekend build fix is going to be. The net gain of this environment is that we have direct exposure to changes as they happen, in real time. We experience the same stuff players will experience as early as humanly possible. It also means that weird crashes and strange one off bugs are more likely to be caught early and the person who added them can fix them when the possible cause of the problem is fresh in their mind.

As the project grows you have to do things to keep a separate version of the game in good working order. So we have a separate version that artists and designers can use to keep working in the event that some developers checks something in and breaks things for a while. However, we still promote every night to keep things nice and synched as well as allowing a “programming” branch to become the artist/designer branch on a whim.

At deployment time, this is incredibly valuable. The system is mostly automated and so deploying a new version of the game requires a minimum of human intervention. By doing this quickly, it means it gets done a lot. By doing it a lot it means that we have a bigger chance of catching all the nasty bugs that make deployment messy and nightmarish. In short, by the time the game launches we have run the patcher, deployed a build, updated the database, upgraded the servers and created the client files literally thousands of times. Even with this process in place we STILL find problems with some frequency. This ties directly back to the premise of iterate to quality. See, I told you it was relevant. :)

MMOs are hideous beasts with complex moving parts that change all the time. It’s kind of like riding a unicycle blindfolded while juggling chainsaws and firing a crossbow at a randomly spinning target.

So how does any of this help you, the player? Well the idea is that by having these processes for development in place we can deliver things to you much more quickly and with greater success. Reliability is not something that MMOs are known for. Not in feature or content deployment, not in scalability, and certainly not in game balance. While some of those problems have difficult solutions we thought we could at least take some of the known risk out of the process and increase the chances that players get something they can have confidence in.

As we go into more public beta testing, the reliability of this kind of system gets pushed to the limit. In public testing is when many of the major problems with MMOs first become exposed. No matter how much unit testing or bot testing you do, there’s something about having a few thousand real people trying all kinds of things that exposes new problems. Part of what we will accomplish with expanding public testing is making sure that the deployment system is tested to ensure a quality experience for customers after a commercial release. This is a result of both frequent and rapid deployment processes.

The other thing I wanted to do with this was to give some credit to the thousands of people in the MMO industry that design, implement and support deployment and build systems. Their jobs are tough. They usually involve working late nights (after builds happen and break), deal with many organizations (development, operations, customer service and billing) and suffer front line exposure when things explode (Oh noes11! The server went down again!!!!). Being the person who sends the “new build is up” email is not fun when the new build causes massive data corruption.

So next time you are playing an MMO that pushes some new content or feature, has a hotfix or deals with some data integrity problems, give a silent thank you to the people in charge of these systems… they live in the reality that when nothing happens they have done a good job and they only hear about their job when disaster strikes.

A game is by definition an audio/visual experience. It is usually the visual side of things that get the most attention and so little time is spent on the other half of the equation. While visuals are critical for having a successful game, having great audio is very important. To think this deeper, try imagining Star Wars without John Williams' sound track or James Earl Jones' booming Darth Vader voice. Poor audio on top of great video can really deteriorate production value. The trouble with audio is that it generally comes late in the development cycle and all to often as an afterthought. Another difficulty is that while poor visuals tend to be obvious, poor audio tends to be a silent failure. Star Wars with slightly worse music would create a sense that "something is missing" or "something isn't right" whereas a bad special effect is much more likely to be spotted for what it is.

So how do we approach audio in a space game like Jumpgate Evolution? The most important thing is to have a kick a** audio department. We have a full in-house audio production team along with good external talent who knows how to “articulate” music. As the visuals and tone of the game begin shaping up we make sure the audio department has access to as much information as possible: concept art, story documents and, of course, a playable version of the experience. We then spend a lot of time looking at other forms of media: movies, tv shows, commercials, other games... anything that might inspire. For a space game like Jumpgate Evolution, it can be quite challenging because there are limited references we can take from. Such as how should alien ships sound when they fly past or what should we use for the environment ambient sounds because in reality, space has no sound.

But our goal is to try to get that same emotional response from the game. Therefore a lot of this is breaking down the various parts that make a given scene evoke an emotional response. By removing elements from a scene you can quickly realize what the core elements of the scene are. For example, does a combat scene in Battlestar Galactica still hold up if the music is different, or if there are no special FX sounds, or if you remove the dialog with some radio distortion. Many things are also very subtle. For example we noticed that in many space scenes there are two levels of spoken audio. The primary is audio directed at the viewer (or the protagonist) and then the secondary audio which is voice chatter between external groups.

The next step is to do some basic concept work. This is done by taking a few minutes of unedited game play video and then letting the sound and music guys do whatever they want to make it sound like the experience we are trying to achieve. This is then evaluated by people watching the video and giving feedback. Once we have a few minutes of game play with the audio the way we want it, we meet with engineers and evaluate the feasibility of the various technical pieces involved. By taking this approach we can avoid spending time on features that will not provide the game experience that people find compelling. We also experiment with existing sound and music to see if it's a quality issue. Sometimes the features and tech are fine, but the audio itself simply requires more iteration.

The final thing to consider is when to stop. As with everything in games, it's difficult to know when you are "there," so we spend a lot of time just testing the game. While we want to create a compelling experience at every level we also don't want to keep working and iterating on things that are starting to have diminishing returns. This point can be incredibly hard to evaluate, especially with something as elusive as audio, but there is a sense that you know it when you hear it.

By having an early prototype that has been tested against a sample audience we can be confident that we are aiming towards a solution that will create the experience players are looking for. By comparing the experience to existing experiences in other popular media, we can be sure that we are measuring against the required quality mark. As with everything else in development it's all about prototyping, rapid iteration and testing.

Most MMOs have some kind of combat system as a major basis for character advancement and progression. Jumpgate is similar in some ways, however as an action MMO, combat tactics are quite different than in other familiar and more RPG like MMOs.

When we approach combat we consider that at the core we must have various roles with both strengths and weaknesses. A dangerous temptation in design is to pick a favorite and overpower it. Avoiding super weapons is a critical part of balance and so the first rule is to design for weakness. As an example, our first two classes of ships were light and heavy fighters. The benefits are reasonably obvious. Light fighters are quick and nimble and heavy fighters are well armored and more heavily armed. The weaknesses are a bit trickier. Should light fighters have less weaponry or less armor or both? Should they have access to less powerful shields and if so, how is this accomplished? Most importantly is how this is communicated to the player and is it actually fun? 

Early on our lead systems designer, Jay Ambrosini, came to the correct conclusion that all of the preliminary balancing was best done in a PvP context. The reasoning is that in PvE, the player needs to feel powerful, but in PvP the fight needs to feel balanced. Once ship classes are balanced in PvP, its not as hard to make the player feel powerful in PvE, but the opposite is not true. We spent many weeks playing just the first class of ship, the light fighter, in teams of 5 or 6 in order to evaluate what it was that made those ships fun to fly and fight. After daily battles, you begin to see what makes those ships work. We also started with the mid level ships as opposed to the low or high level ships. This is primarily because you can find the center point and then work upwards and downwards from there.

With just one class of ship tactics are the easiest to work out. It's mostly people figuring out how to effectively fly inside the map, find cover, maneuver through tight areas and avoid missile lock-ons. It took a lot of time to get all of the various pieces working correctly, but eventually we were having a lot of fun and looking forward to the daily play tests. It wasn't until this point that we added the second class of ships which were the heavy fighters. Jay decided quite early that he wanted the classes to feel VERY different as opposed to having lots of small differences. Thus, the heavy fighters had considerably more armor, came armed with mortar AOE weapons, but were much slower at turning and accelerating. Initially this lead to heavy fighters being almost indestructible at long range and nearly useless at close range. This proved to be quite frustrating as neither side felt they could respond once the condition for victory had been set. Specifically, light fighters had no chance at range if they didn't see the heavy coming in, and heavy fighters were unable to break a fight once the lights got in close. We tried a number of things but what helped a lot was adding a combat boost consumable, mines and chaff. Chaff allowed the light fighters to overcome the heavy missile fire while combat boost allows the heavies a chance to get some distance. A tactic that worked quite well was to mix a few lights with a heavy so that the lights can chase off and incoming enemies and let the heavies stay further back and fire devastating missile and mortar fire.

A third class of ship was then added which we decided should be a more nimble and less armored fighters. One of the PvP maps is a traditional capture and hold scenario and thus rushing to the capture point is a key tactic. Also, being able to zoom around the other fighting ships to draw fire proved to be a very useful tactic. As with the light and heavy fighters it took many weeks of iteration to get the basics of the class defined.

The other thing that was interesting is that many times you have to give a change a while to see if it is good. After many months of playing, some people on the team become quite proficient at playing in very advanced ways; small changes in weapon damage, range, ship speed and anything else. It's difficult to decide between reacting to feedback from a change and having the patience to wait and see if the initial feedback changes after a few days of getting used to it. An example of this is that we changed the missile lock system to take into account line of sight. This was initially a huge disadvantage for heavy fighters and met with some initial criticism. After some time, however, we determined that it was the right decision.

I think that the key thing to understand is that tactics are the result of distinct and strong variation, simple rules and easy to understand advantages and disadvantages. Once you have that, people use that information to create very complex and interesting combinations.

It's very tempting to just throw a bunch of classes of ships together in order to say things like “our game has 15 classes of ships!” but this, we believe, is the wrong direction. People want meaningful and strong choices and not lots of meaningless, empty choices. Currently we plan to have 4-6 classes, but they will each have nearly endless possible configurations within those groups. I also suspect we will continue to add more classes and equipment which will continue the balanced and varied tactics available. Much of this will come from more extensive testing as many thousands of players will provide much better information than our group of developers working alone can.

In Jumpgate Evolution, ship design is one of the most critical components in the game. When we first started working on it, we spent a lot of time with just one ship, in one sector, fighting a handful of enemies. At first we just wanted to make a ship that was both fun and intuitive to fly. This involved changing camera controls, controller input and lots and lots of unit tests. In the end it took about sixty to seventy iterations to get a flight model and control scheme that felt right.

The next thing we did is come up with what kinds of roles we wanted people to have. This was fairly simple in that the kinds of ships that people want are reasonably well defined. You want to have a quick and nimble fighter, a heavier combat ship, perhaps a missile boat of some kind. On the commercial side you need mining and transport ships. For the high end we considered things like “spy” class ships or heavy gunboats, support class ships that can repair armor or refuel ammunition and just about anything in between. It’s a really fun process to just throw ideas out and see what seems cool and fun to try.
 

After that we culled those ideas down to a much smaller number. We wanted to make sure that people could play many types of ships so it was not like choosing a “class” in a fantasy game, for example. Thus one pilot can own a mining vessel, a transport and a heavy fighter. We set to work doing broad strokes on the various classes. Initially this meant that we needed support for the various characteristics a ship might have, for example power, engine thrust (acceleration), mass, turning capability and so on.
 

Our first pass was a multitude of ships with different capabilities and small to moderate differences between them. For example, initially we had shuttles, light fighters, medium fighters and heavy fighters with distinctions and progression within each class. When our systems designer started working on implementing all of those he quickly came to the conclusion that lost of small differences aren’t very fun. We thus switched things around to make much stronger and clear distinctions between each class of ship. We ditched the idea of a medium fighter and took what was cool about it and split it between light and heavy. Suddenly it was much more distinct and fun to fly the two different classes. At first this decision hurt a bit as we thought of it as removing variety from the game. The opposite turned out to be the case. Variety comes from the ability to distinguish difference. If everything is only slightly different than something else, it turns things into a kind of gray goopy mess. By strongly identifying differences flying a light fighter, a heavy fighter, and an industrial ship feels right.
 

That’s a pretty general overview of one particular aspect of what goes into working on ships. It also involves balancing power availability, speed and maneuvering, equipment distinction and many other things. Another thing we learned is to balance in PVP first. When we make new ships now, Jay (Jay Ambrosini – Systems Designer) will spend quite a bit of time taking a stab at how things should fly, what configuration they should have and so on. We then give people those ships and during our daily play test do a series of PVP battles followed by everyone sending Jay feedback. He takes that, makes some changes and we play again. Sometimes we do this up to three or four times a day. You know when you’ve got it right because people won’t want to stop playing. Once a ship works in the PVP scenario, tuning it for PVE is much easier. Jay said it best when he proclaimed that PVE has to be fun, but PVP has to be fair. Making sure ships are distinct but avoiding super weapons is quite difficult. Our overall goal for ships is that there are as many distinct types as possible and that they are all enjoyable to play while each having weaknesses that can be exploited. This is how, we believe, you can achieve the kind of tit for tat combat that are a hallmark of great game design.