Sanya you said "an MMO can be flawlessly designed with intricacies like a mechanical Swiss watch and still suck the chrome off a bumper when those intricacies collide with a large number of human beings. "

You seem to be saying that something that "sucks the chrome off a bumper" is a bad thing when it is actually a very good thing.

Part of the problem isn't just the players, it is the way beta is managed by the development team as well. Most of the betas I have been invited to over the last few years have said, "Go ahead and play the game and see what doesn't work." This is very open-ended, non-focused testing and it is very easy for players to assume systems and mechanics are largely complete.

Years prior, I remember entering betas where they were conducted in phases which focused on functional testing of certain, specific aspects of the game. Sometimes this limited individuals to a certain race/class combination, sometimes a specific level range. This gave the feeling that the development team understood where certain systems were but needed dedicated feedback on other areas. This created a much more concentrated experience, and not just of the first 1-20 levels (which seems to be the common beta restriction today).

Improve the structure of your beta, and you will improve the feedback and buy-in of the players involved in it.

More than a few games have had successful small betas and then fallen on their face at launch because they never did any stress testing. A proper stress test takes thousands and possibly tens of thousands of players online simultaneously. That is really only achieved in large scale open beta.

Originally posted by AcmeGamer

  Good article and I agree across the board. Beta is now a useless PR tool. Smoke and mirrors by and large. I won't even bother to sign up for betas anymore.

 

Most of the games now are so simplistic they in reality have no complex systems that really need testing. The genre is pretty much dead for anyone with more than an IQ of 1.

My simple response to this is that you don't invite people randomly if you want testers.

I've been in MMO betas since as far back as UO, but these days because of the random invites I rarely get in until a few weeks before launch. I'd happily test and accept the beta for what it is, if I got invited.

But hey, random invites...

Open beta = Free trial right?

Well, that's all they are today, no more real testing taking place by time they let folks come in. (stress test is probably a more accurate term).

I mean look at all the open beta testing SWTOR launched with and yet it managed to launch with some issues regardless. (though I blame that mostly on a rushed launch)

Here is how I would execute a beta test for a game I'm developing. I'd canvas for beta testers as normal, but I wouldn't have beta forums. At all.  Beta Testers would be sorted into teams, or cells. Some times they'd be given no instruction, and allowed just to play. Other times each cell would be given a task to test.

This would help mitigate the echo chamber effect that beta forums suffer from. You get the most vocal minority posting at times, thus influencing decisions of others.

Also, the cell approach would reduce the effect of those in the beta who are playing for a peek. You would assign cells only partial access to the game, but by having many cells, you could cover the entire expanse of the game.

Originally posted by heavyhebrew

OP forgot one salient point about beta testing. Those who do beta test and submit bugs, glaring problems like memory leaks and comment on bad design and are ignored.

 Me being one of them .. whenever I can i'll beta test a game and i'm used to bugs that are reported making there way into the release but the saddest thing is the players in beta's these days :-(

You get a good deal of them being just there to try the game out, the amount of questions like 'will my character be wiped' and then the moaning and complaining when told ofcourse it would be.

Funnily they always know an mmo that they beta'd and their toon wasn't wiped .. out of every one i've done (and this is going back a bit .. lol) I haven't known one to do that.

As someone above said, people are using beta's now as free trial's because they don't want to pay for the game to find out if they would like it or wait for the free trials to be released.

Instead they'll just pose as testers and some will end up slagging the game off even though they were testing an unfinished product (ok .. mmo's are basically never finished but you get what i mean)

 

I was in the beta for vanguard for an entire YEAR...  I also rose to being the tailoring crafter liasion. 

Personally I think beta should be earned not bought.  They should call those late beta megatests what they really are "Stress and Tuning Pre-Play." 

A real beta gives you actual access to the devs or at least one layer removed from the devs where your feedback is actually worthwhile.  They play with the people -- I remember Aradunes thestwian wanger...  This kidding around takes some of the edge off... 

A real beta test should REQUIRE a commitment to finding bugs.  If you are just playing the game as a preview and do not find anything you should be bumped for people who are more willing to spend the time to actually make the game better.

I am one of the people who actually find real bugs even in a tuning pre-play -- I know I found my share in rift and that was thought of as a pretty bug-free experience by most.  You have to mess with the games a little sometimes to make them give up their mistakes.  Try doing things most people wont do but some people could do. 

With the lengthening of the development cycles for games, I do not see why they couldnt have medium sized (say 3 servers) beta tests significantly earlier in the development cycle than we see today.  Let people see the game at a point where they can actually make a difference and find more bugs than geometry issues and typos.  Encourage people to report on things that are unbalancing as well.. 

These days a lot of people will actually HIDE an exploit they find during beta rather than reporting it, hoping to exploit it later.  Finding an exploit class bug and not reporting it should be worth a ban from not only that beta but any other beta run by the company in the future.  Looking for and reporting exploit-class bugs is one of the most important thing you can do as a beta-tester.  I know I found a few in games. 

I also think there should be rewards for early beta testing.  Or even late beta testing if you manage to distinguish yourself.  I mean if you find and turn in an exploit-class bug you should at the very least get a free copy of the game and an attaboy.  They should keep lists of people who have found and report even things like typos etc so they get into future beta tests earlier rather than later.

If when the game released you got some in game benefit for finding that typo in the second to last beta phase that somehow made it through that far and then was fixed before release, people would treat the betas with more respect.

Originally posted by centkin
 

I also think there should be rewards for early beta testing.  Or even late beta testing if you manage to distinguish yourself.  I mean if you find and turn in an exploit-class bug you should at the very least get a free copy of the game and an attaboy.  They should keep lists of people who have found and report even things like typos etc so they get into future beta tests earlier rather than later.

If when the game released you got some in game benefit for finding that typo in the second to last beta phase that somehow made it through that far and then was fixed before release, people would treat the betas with more respect.

This is actually a pretty good idea. Atm there are rewards for players who beta "test" and find an exploit and don't report it... they can use that exploit at release and gain xp/money from it.

Perhaps giving free copies of the game away is abit much though, maybe giving ingame items that scale up depending on the amount or severity of the bug/s found would be incentive enough. Nothing that would give a massive advvantage to a player, but maybe smth like:

LVL1: title

LVL2: LVL1 + pet

LVL3: LVL2 + mount

LVL4: LVL3 + ingame item to boost xp for set duration

That being said, its a sad reflection of the current state of many gamers that rewards for finding and reporting bugs are even being mentioned :(

Originally posted by Waizer

Originally posted by centkin
 

I also think there should be rewards for early beta testing.  Or even late beta testing if you manage to distinguish yourself.  I mean if you find and turn in an exploit-class bug you should at the very least get a free copy of the game and an attaboy.  They should keep lists of people who have found and report even things like typos etc so they get into future beta tests earlier rather than later.

If when the game released you got some in game benefit for finding that typo in the second to last beta phase that somehow made it through that far and then was fixed before release, people would treat the betas with more respect.

This is actually a pretty good idea. Atm there are rewards for players who beta "test" and find an exploit and don't report it... they can use that exploit at release and gain xp/money from it.

 

Perhaps giving free copies of the game away is abit much though, maybe giving ingame items that scale up depending on the amount or severity of the bug/s found would be incentive enough. Nothing that would give a massive advvantage to a player, but maybe smth like:

LVL1: title

LVL2: LVL1 + pet

LVL3: LVL2 + mount

LVL4: LVL3 + ingame item to boost xp for set duration

 

That being said, its a sad reflection of the current state of many gamers that rewards for finding and reporting bugs are even being mentioned :(

Considering that people shouldnt be finding tons of exploitable level bugs, giving a free copy of the game is not a large issue -- especially because a free digital copy really doesnt cost the company anything.  I say this because lets say you played a game in beta and you found a major issue in the game and you were rewarded with a free copy....  The odds that you are going to stay with that game for a few months is MUCH higher than if you reported a major bug and got the typical "We appreciate your blah blah" form letter. 

It is also a form of viral publicity.  Instead of potentially having someone out there saying "Yeah the game is a buggy mess," you have someone out there saying "This company cares enough about fixing their game that they gave me a free copy."  Lastly that the possibility is out there would make more bugs be reported which is a very good thing.

>>How do I, as a developer, convince you that beta means BETA?

Make Beta Testing unpleasent.

Drive away the hordes that are just there for a 'free game' or a 'head start'.

Do this by limiting the options of the testers.

Give your 'testers' limitied scope within the game.

For example:

Some are limited to just character generation.  They can generate a character...and another...and another...but they cannot actually play.

Some can play the tutorial.  They are assigned a character (pre created) and they can play the tutorial levels and zones... then they can 're-roll' and do that again.... and again... and again...

Have some that can play only level 1-10...some that can play 11-20... some that can only play certain zones... oh...and did I mention the weekly character wipes?

The reward is that for those who stick around... find and report bugs... test the boring stuff...actually contribute in a meaningful way on the (beta) forums... after a while they get greater 'freedom'.  They get to Create their own characters and level them up all the way.  Maybe they are still locked to a class or faction (that you want tested) but it's better.

And the ones that really stick it out and are of value - get free run in the end.  But it takes time.  It takes HARD WORK.

Maybe keep a register of the best testers - maybe it's something the industry as a whole needs to do?

Perhaps a forum run by developers or even a sub forum on a well known site like MMORPG.com where members can only access it if they can produce a log in code supplied by appreciative Devs?

And membership lapses after a few months. 

That way you have a pool of real testers available that you know have been of real value to other devs.  Not fool proof - but better than the mouth breathers that you get by giving away 'keys'.

This has been a problem for years:

http://www.mmorpg.com/discussion2.cfm/thread/209376/page/1

If you want to run 'pre release promotions' then call it that.  NOT a "Beta Test"

If you want to run a 'stress test to failure' then call it that.  NOT a "Beta Test"

And at the end of a Beta Test...wipe all characters without exception.  Make it clear this will happen.  Those that choose to leave as a result are not 'testers' anyway.

Note: I have 'Beta Tested' several games now.  I actually report bugs and set out to break things.  In one MMO I managed to break the Tutorial less than a fortnight prior to retail release.  I was left wondering exacly what the other 'testers' had done for the previous months?

I have been personally invited back (they sent me an email) by one company to test an expansion.  In that game I had been extremely critical of the game in the Beta forum but I had always given a reason and proposed solutions.  I was attacked by many Fanbois for my comments and most of my comments resulted in no change to the game - BUT - apparently the Devs liked having someone around who was not a 'yes man'?

Oh No  Sanya, you are starting to sound like me! -)

I can still remember when Devs required actual formatted reports to be sent via email when testing.  Or when characters were wiped every 3 days.  Or the server(s) were reset every 25 minutes.  And us testers loved it all.

Testing DAoC was like being included in a large family.  Everyone listened to each other and testers helped the Devs and each other.  The bonding community made you care about the project.  I found this to be true when testing F&F for LOTRO:SoA... especially with Tiggs and her Extra Crispy guild involved.  Testers felt like they were assets to the projects...not just another warm body to run around in the game while background-game-logging programs tracked our playstyles.

Testing has gone from individual input to spreadsheet input.  We are now ants keeping the hive humming while our actions are being tracked and quantified in automated reporting tools.  And testers are no longer given the tools to help report locations and issues.  Recent games I've tested (and one I am currently testing) don't have a /loc command, no way to take screenshot in-game (other than using Fraps), no interaction with the Devs/Mods/CSRs, bug reports only receive auto-generated replies (Tabula Rasa had a great bug reporting system!), and in-game help tickets are ignored & deleted by GMs with no reasons given.

LOTRO has an invite-only test server where new content is tested prior to release to the public test server.  Turbine Devs interact directly with these testers to break new systems, flesh out new content, listen to ideas, and focus-test team-based areas.  I am sure other game publishers & developers have servers like Turbine's but just don't publish the fact.

I sincerely hope game companies return to the idea that testers are there to help you create a stable enjoyable enviroment and are assets who want to invest their time FREE OF CHARGE to help the projects.... and not just another person who might buy the game if they get pre-release exposure.

TQQdles™,

Dolnor Numbwit

aka Noldor Numbwit

Eternal Newbie

P.s. I still wear the DAoC Road Trip T-shirt now and then! -) 

1) Define exactly what you are looking to do, both for yourselves and your audience.

2) Don't call a stress test or prerelease test drive a 'beta'.

3) Pick your first rounds of beta testers carefully. It isn't that hard... just look for the folks on your forums who take the time to wrote out thoughtful, logical topics or replies. Invite them. Those people are in turn likely to know people who wouldn't be a waste of time. Let your proven testers invite 5 more people. Simple.

4) Make certain that the bug handling process is as transparent as possible. Feedback to the community is MANDATORY. As others have explained earlier in this thread, nothing discredits developers and a company more than having issues reported, ignored, then left unresolved for long periods accompanied by a communications blackout. That type of behavior is common, and sends a tremendously negative set of messages to the community.