Talking About Closed Beta

MMORPG.com:

Can you tell us a little bit about the structure of Star Trek Online's beta process?

Daniel Stahl:

While employees and our friends and family have been playing the game for the last year, Star Trek online started the Beta process this last fall by inviting players to help us with focused weekly play tests. We would prepare a new patch of content, and then ask players to focus on a specific area of testing to give us quality feedback on the system under scrutiny.

In addition we have been monitoring the technology behind the game making detailed changes to how our servers perform. Now that we've passed some of our technical hurdles, we are opening up the content in the game and letting players progress as far as possible and give feedback.

MMORPG.com:

Who is currently in the STO closed beta and how have you decided thus far who to include?

Daniel Stahl:

Employees, our friends and family, Champions Online lifetime and 6 month subscribers, press members, contest winners, publishing partners, and web applicants are all in Closed Beta at this time.

Prior to the start of closed beta, employees have been playing the game alongside our friends and family members until we felt the game was at a point we could invite our Beta applicants. As we've increased the population cap on our testing shard, we've given beta access priority to Champions and Lifetime 6 month subscribers first, based on their purchase date. We are continuing to invite web applicants and looking forward to Open Beta in January.

MMORPG.com:

Is there any kind of intentional balance between experienced MMO players and non-MMO Trek fans in the current makeup of your testers?

Daniel Stahl:

We have not limited who is in Beta based on their prior game experience or their level of Trek knowledge. Luckily there is a healthy interest in Star Trek Online from many types of players and if the Beta Forums are any indicator, we have a good mix of players providing feedback on different parts of the game.

MMORPG.com:

Can you speak to what happened regarding beta invitations for players who were granted STO beta access through Champions Online subscriptions?

Daniel Stahl:

When we started Beta this last fall, our server had a set population capacity that we've had to stick to in order to provide a decent gameplay experience for those play testing. We started with inviting Champions Lifetime and 6 month subscribers based on the order they had subscribed.

As server capacity increased and our back end technology improved, we added more players from the list each week and as of last week we have sent Beta invites to all players in that group. If any Champions Lifetime or 6 month subscribers have yet to receive their beta invite, they should contact our customer service departing immediately.

MMORPG.com:

Can you give us a few examples of issues that were caught due to beta feedback that have since been changed or re-vamped?

Daniel Stahl:

There are plenty of examples! Here's an example where the Beta community was instrumental in helping us make needed changes.

Ground Combat recently underwent an overhaul based on Beta feedback. The players felt that ground combat was too fast and needed more tactical depth. They also wanted to be able to have greater control over their away team. The design team read through the many threads and came up with a list of changes that made sense and rolled it out to Beta. Ground Combat is now greatly improved thanks to this.

There are other similar examples related to combat balance, level progression, and overall feel of some missions. Our Beta testers have been awesome at getting us feedback.

MMORPG.com:

How do you, as developers, decide what to act on in terms of beta feedback and what not to?

Daniel Stahl:

The team doesn't make knee-jerk reactions just because a player has feedback on something. Instead we read the discussions and look at our designs to determine if there needs to be changes based on what the players are saying. When we look closely we often find common themes surfacing across multiple discussions. Instead of taking each bit of feedback as a bug, we look at the cause and implication of what needs to be dealt with and act appropriately.

For example, some players may complain that enemy A is too tough, while other players may say mission B is too easy. When we look at the issues in a holistic way, we may determine that the real issue is a problem with player item C.

When it comes to more subjective feedback, we look for consensus and evaluate against our design goals to ensure we are making the changes players want in a thoughtful way.

MMORPG.com:

How important is the beta process to the design of Star Trek Online?

Daniel Stahl:

Beta has been critical for helping us validate our designs. Internally we can evaluate our data and argue over how long it should take to do various task, or draw up charts of how specific items should interact. That sort of data can only go so far in theory.

Once you allow players to go through the content and play game from start to finish, you find things you never would have considered simply because players do the unexpected. Beta helps the designers find all of these edge cases.

MMORPG.com:

How important is the closed beta stage as opposed to, say the open beta stage that comes just prior to launch?

Daniel Stahl:

Closed beta provides a canvas that can be quickly iterated on so that we can quickly react to feedback and fix up game breaking issues before we let in the masses. It is by nature less stable, but also the more fluid that Open Beta. We can patch at will and make sweeping changes to that may even require a character wipe.

Open Betas are where we test the launch capability of the server technology. It is the trial period for the back end hardware and patching system. It also gives us a good idea where we need to focus efforts after launch.

Both stages are equally important even though they may have different objectives.

MMORPG.com:

Often, beta communities are the best judges as to whether or not a game is being rushed out the door. How did your beta community react to the announcement of a February 2nd, 2010 launch date?

Daniel Stahl:

Once a game launches, I could see where in hindsight that may be true. However, when you are in Closed Beta pushing constant patches to the testers, the feedback can vary a great deal from build to build. There are some patches where testers are as happy as a Tribble and would buy the game right then and there, and conversely there are some patches where they want to throw their monitors through the window like an angry Gorn due to some bug we introduced.

Ultimately it is the responsibility of the STO team that we stay on schedule and deliver a quality game that fans will enjoy. We feel very confident in our game and the entire team is working round the clock to ensure that every patch we make to Beta players makes the game better. The most meaningful indicator we read on the forums is that Beta players find that we are responsive and listening to their feedback. We'll continue to foster that relationship all the way up through launch and beyond.