After much deliberation I decided to devote this blog to one of the unsung heroes of MMOs, which is the developers that are in-charge of building and deployment. Now before you all check-out over what you think sounds like a boring topic, I want to remind everyone that without a decent build and deployment system, no one will be able to play our game.
As you guys have heard many times by now, Jumpgate is built on the concept that iteration is a central requirement for quality. This begins with the build and deployment process. I’ve worked on lots of software projects where getting a build together was a nightmare… getting it out the door? Ha! Almost impossible.
Here’s how things normally work on such projects. Everyone is kind of working on their own stuff, on their own machine; chugging along, making awesome progress. When they feel that it’s at a functioning state they check it into whatever version control software they are using. At some point (usually every 2 or 4 weeks) “build day” comes along. It is at this time that all these wonderful changes everyone made come together and…. BOOM the whole thing explodes. This is almost always on a Friday, or even better, just before a holiday, which results in people spending late nights and/or weekends trying to remember everything they did for the last 2-4 weeks that might have caused the explosion. It’s lots of fun!
On JGE we set out to avoid this kind of thing. From the beginning of the project we setup a system by which all changes are made in a central location. Once a change is detected our build machine automatically makes a build of the game and sends out all the change notes to all the developers. The next time someone runs the patcher (yes our internal process uses the same patcher that we distribute), they are seeing those changes, in most cases minutes after they were made. This means we have more frequent, easier to solve, smaller explosions along the way and, eventually, very few explosions. It also means that we have to keep our build times down to about 10-15 minutes. I’ve heard of MMOs that take an entire day to build. Imagine how fun that weekend build fix is going to be. The net gain of this environment is that we have direct exposure to changes as they happen, in real time. We experience the same stuff players will experience as early as humanly possible. It also means that weird crashes and strange one off bugs are more likely to be caught early and the person who added them can fix them when the possible cause of the problem is fresh in their mind.
As the project grows you have to do things to keep a separate version of the game in good working order. So we have a separate version that artists and designers can use to keep working in the event that some developers checks something in and breaks things for a while. However, we still promote every night to keep things nice and synched as well as allowing a “programming” branch to become the artist/designer branch on a whim.
At deployment time, this is incredibly valuable. The system is mostly automated and so deploying a new version of the game requires a minimum of human intervention. By doing this quickly, it means it gets done a lot. By doing it a lot it means that we have a bigger chance of catching all the nasty bugs that make deployment messy and nightmarish. In short, by the time the game launches we have run the patcher, deployed a build, updated the database, upgraded the servers and created the client files literally thousands of times. Even with this process in place we STILL find problems with some frequency. This ties directly back to the premise of iterate to quality. See, I told you it was relevant. :)
MMOs are hideous beasts with complex moving parts that change all the time. It’s kind of like riding a unicycle blindfolded while juggling chainsaws and firing a crossbow at a randomly spinning target.
So how does any of this help you, the player? Well the idea is that by having these processes for development in place we can deliver things to you much more quickly and with greater success. Reliability is not something that MMOs are known for. Not in feature or content deployment, not in scalability, and certainly not in game balance. While some of those problems have difficult solutions we thought we could at least take some of the known risk out of the process and increase the chances that players get something they can have confidence in.
As we go into more public beta testing, the reliability of this kind of system gets pushed to the limit. In public testing is when many of the major problems with MMOs first become exposed. No matter how much unit testing or bot testing you do, there’s something about having a few thousand real people trying all kinds of things that exposes new problems. Part of what we will accomplish with expanding public testing is making sure that the deployment system is tested to ensure a quality experience for customers after a commercial release. This is a result of both frequent and rapid deployment processes.
The other thing I wanted to do with this was to give some credit to the thousands of people in the MMO industry that design, implement and support deployment and build systems. Their jobs are tough. They usually involve working late nights (after builds happen and break), deal with many organizations (development, operations, customer service and billing) and suffer front line exposure when things explode (Oh noes11! The server went down again!!!!). Being the person who sends the “new build is up” email is not fun when the new build causes massive data corruption.
So next time you are playing an MMO that pushes some new content or feature, has a hotfix or deals with some data integrity problems, give a silent thank you to the people in charge of these systems… they live in the reality that when nothing happens they have done a good job and they only hear about their job when disaster strikes.