Life with Beta

Pirates of the Burning Sea Producer, John Scott Tynes, describes his "Life with Beta" in this newest developer journal.

Operating an MMO beta is fantastic. You get all the fun of interacting with real players who are really playing your game, but you're still below the radar enough that the pressure isn't overwhelming. It's a great opportunity to learn what it's going to be like to run a live service.

We started our beta program a year ago, in late 2005. Players who tried the game back then and then tried it again more recently are amazed at how much has changed. But that's what a year of development gets you! The game has come a long, long way in that year and is much better than ever before.

This past December, just before the holidays, we made the big jump to 24/7 operation. Prior to this, we were opening the servers on Tuesday, Wednesday, and Thursday evenings, and also on Saturday and Sunday mornings. We kept the times restricted so that our testers wouldn't burn out too soon and also so we could have GMs in game to solve problems.

Happily, though, we just don't have many problems that require GM intervention. And now that we're about to release the level 31-50 mission content to the beta, we're not so worried about burnout. Plus, we really wanted our testers to fully engage the player-driven economy, and that means they need more frequent access so the production and auction systems can really work.

So, we rolled out a new build, turned on the servers, and left the office for the Christmas holidays.

Oops! We didn't really talk through the 24/7 switch as thoroughly as we should, because the beta has really been pretty problem-free. What happened next was a great reminder of how transitioning from product development to a service-oriented mindset can be a challenge. But that's why they call it a beta . . .

The new build introduced a bug we didn't catch in our testbed session, which we run prior to releasing a new build to the full beta. One of our servers was taking an unusually long time to start up, long enough that our automated cluster controller was killing it and starting a replacement. This meant it couldn't start up successfully without manual intervention after the daily reboot.

With our staff scattered to the four winds, it became a game of hot potato every day to ensure that someone could remote desktop in to the beta cluster and kick the server into starting up. For a couple of days I would check the beta forums at some idle moment, expecting to read about happy players sailing along, and find that once again we hadn't dealt with the server startup. Argh!

Fortunately we got a handle on the issue and applied a fix to the server startup problem. But it made for a couple of exciting days right in the middle of opening presents and making Christmas dinner for the family.

I gathered some statistics about what our beta testers were doing before the switch to 24/7 and after to compare the effect that had on the beta. Here are some numbers you might find interesting about what happened after we switched to 24/7, based on a comparison between two three-day periods before and after the switch:

• Broadsides fired: up 44%
• Missions completed: up 78%
• Recipes used: up 14%
• Auction listings created: up 31%
• Auction items purchased: up 61%

The best news for our testing purposes here is that while participation in the production side of the economy went up slightly, participation in purchasing player-made goods went up hugely. We thought moving to a 24/7 schedule would greatly improve the auction system and sure enough, it did.

Anyway, that's the word this month from the Burning Sea. We're six months until launch and full steam ahead!

- John Scott Tynes
Producer, Pirates of the Burning Sea