Interessting whine. But I'm afraid it's one sided.

If you leave hype aside beta does serve a good purpose if enough people join. It gives devs an opportunity to check the following:

- Performance: A small "elite" group of testers can only focus on functionality of the game. Never on performance and load. Never! Just ask yourself what good is a perfectly polished game that crashes after 100 people log in. And lets not forget ping (lag). A selected group of testers would probably be local, maybe even hired by the company. But if you pass enough beta keys around and you get people loging in from various locations and countries using differens isps then you have a good chance catching problems you never dream existed. This is what performance and load are about. There are also subsections like login server performance, or authentication server performance (usually separate from the main game server) but this is more or less all in same direction. The more people you get the more data you can collect about how your game behaves in extreme cases.

- Functional bugs: Take a good look at your calculator and then try to think about number of options one can use to enter a simple equation. Amazing how many "test cases" can you get out of a simple calculator, right? Now take a look at any game and think about nuber of options a player has to go through, starting with login, avatar creation screen, naming, tutorial... Bazillion gazillion combinations in an average game. This can not be covered by a test team or by small group of 1337 beta testers. The more users you get on your server the bigger the probability is that one of them will discover a way to break the game. It is a probability game, nothing more, nothing less.

- "Idiot users": Let's be honest. All devs can focus on is the positives! After more then a decade in software development this is an axiom in my opinion. A fact! No developer ever assumes that some users might even think about using his application (a game in this case)  to do something that was never planned with it. It's like trying to water the plants with a chainsaw or trying to light a fire with a hammer. Maybe it can be done but results would probably be devastating for plants and trees. Same with "idiot users", who can behave "in the wrong way" simply because their assumptions are wrong or their knowledge of gaming and technology is very poor. It's like the story of a lady that used CD tray as a cup holder or of another user that used a mouse by trying to step on it. Small group of beta testers would NEVER have anyone in that group. And if game's error handling routines are poor and an "idiot" user keeps doing something silly that constantly crashes the server then EVERYONE will suffer and not just him.

 - "1337 users": Opposite of "idiots". Those guys can hack into your game and cause havoc for anyone around. Botting, scripting, even using APIs to make your own apps and screw the game in your advantage. Good luck trying to find all security holes by letting 100 loyal testers do the testing. And how many games you know that died due to hacking and boting?

There are other things that could be said in gavor of current beta practices. But the way current gaming industry is set up there is NO WAY around how beta is done. Pass quantillion keys and hope for the best. Yes there is also greed involved when companies release unfinished products since all of the guys in the industry want to make money (I would be worried if any of them dont). Greed also plays a big role when you don't want to have proper test teams, which is not problem since you can always find million souls just waiting to log in and try something new. So public gets something in return for early access.

All of the above is "imho". Yet my "ho" is based on a decade of software testing experience and more then 2 decades of gaming experience so I think I know what I'm talking about.:)

Right after reading this i ended up on the TERA beta signup page.

http://tera.enmasse.com/news/posts/beta-test-schedule

After reading that, noone can say you are wrong :) ... not a word about "improving the game"  or " finding bugs" ... what has the emphasis is "Eager to get your hands on TERA?"

1. Never use beta as a promotion tool splashed all over the web page, pre-order and get a spot in the “beta test”
2. Do not do a twitter/facebook contest to get into “beta”.  Matter of fact; do not use Twitter or Facebook at all for this.   If you are looking for real beta testers those are the LAST places you should going.
3. Be very clear what stage the game is in, what the beta tester can expect from the game in its current state and very specifically what testing you are doing and feed back you are requiring from them.  Yes I said it REQUIRING from them.  This is not a play test make sure that is completely understood.
4. Once you do let us in, do not then turn around and tell us we do not know what we are talking when we give specific feedback you do not want to hear.  If a very large majority of the real beta testers tells you something in very specific detail without trolling is VERY broken in your game, listen to them do not threaten them or out right dismiss them. 

Originally posted by MarlonB

Right after reading this i ended up on the TERA beta signup page.

http://tera.enmasse.com/news/posts/beta-test-schedule

 

After reading that, noone can say you are wrong :) ... not a word about "improving the game"  or " finding bugs" ... what has the emphasis is "Eager to get your hands on TERA?"

 

And still servers went down which only confirms my theory that this is the right way to go. They let bazillion players try to log in which gave them perfect load test. Beta test successful. The facts that no one could play is irrelevant for devs. They will sell it to you as "early sneak peak" when in fact it's "early load test".

I'm not really sure a basic language port sneak peak really constitutes a beta.

Each company runs betas differently. The server load/mass invite beta is generally a final stage of beta. Recent betas like Rift and SWToR were sneak peaks. Hell in SWToR they even disabled the ingame bug reporting.

Originally posted by Spiider

Originally posted by MarlonB

Right after reading this i ended up on the TERA beta signup page.

http://tera.enmasse.com/news/posts/beta-test-schedule

 

After reading that, noone can say you are wrong :) ... not a word about "improving the game"  or " finding bugs" ... what has the emphasis is "Eager to get your hands on TERA?"

 

And still servers went down which only confirms my theory that this is the right way to go. They let bazillion players try to log in which gave them perfect load test. Beta test successful. The facts that no one could play is irrelevant for devs. They will sell it to you as "early sneak peak" when in fact it's "early load test".

 

I can follow that logic ... but this is not one test .. it's for all (5) closed beta tests.

I decided to come back to this thread to add something.

Beyond the promontional use of beta and the various points made in the article.   There is another common thing that pops up in betas now.

Early on in the TOR general test when the population was still quite small...   A very vocal group of testers spent a lot of time trying to totally redesign the game.   I've seen this in other tests of course but this was the most recent.

Somehow it seems people miss the point that they want us to test the game they are making.   As opposed to trying to tell them the game we (or some of us) want them to make.   In fact I would say there were more threads on the beta forums arguing about the game that wasn't being made (any apsect of MMO you could come up with that were not going to be in TOR) than threads discussing issues with the game we actually were testing.

You can always invite a large group of people and toss in the promotional crap when you need to load test your servers.   Perhaps developers should be more active with their test community in removing testers who are only devoting time to ... the game that isn't instead of testing the one there is.   It would also clean up their beta forums so people could more easily find relevant threads to their testing issues etc

Originally posted by Antarious

Somehow it seems people miss the point that they want us to test the game they are making.   As opposed to trying to tell them the game we (or some of us) want them to make.  

 

*hangs head in shame*

But in defence of all the armchair developers out there, is this really a bad thing?  

I realize that critiquing the core design in a fully realized beta is a little pointless and so is appointing yourself excutive producer.  But if you have sincere high-level feedback and you haven't been told what features are written in stone, what's the point of keeping it to yourself?

To me the problem is all the previous games that were beta tested came out with the same exact bugs reported in beta. IE. Vanguard, AOC, WAR, the long list of games that need to be patched after retail and others. This long storied tradition has made people lose faith in beta testing, and see it for what it really is. A marketing ploy where feedback and bug reports are ignored or shelved for after release.

Now, whenever a dev team wants to actually get bug reports they have to deal with what the industry has created - Gamers that don't believe bug reports are taken seriously. People can only be fooled so many times before they catch on and adapt.

Originally posted by SBFord
How do I, as a developer, convince you that beta means BETA?

Do not use the word beta at all. When asked, deny you are looking for beta testers but say that you are looking for people to really test your game. The word beta should be considered to be contaminated in the gaming industry.

Originally posted by Astraeis

Do not use the word beta at all. When asked, deny you are looking for beta testers but say that you are looking for people to really test your game. The word beta should be considered to be contaminated in the gaming industry.

 

Or how about players go with what the devs are looking for and actually test the product rather than just use it for a sneak preview? The players are just as complicite... at least from my experience of watching the Rift and ToR boards on here during their open betas...

Originally posted by MurlockDance

Originally posted by Astraeis

Originally posted by SBFord
How do I, as a developer, convince you that beta means BETA?

Do not use the word beta at all. When asked, deny you are looking for beta testers but say that you are looking for people to really test your game. The word beta should be considered to be contaminated in the gaming industry.

Or how about players go with what the devs are looking for and actually test the product rather than just use it for a sneak preview? The players are just as complicite... at least from my experience of watching the Rift and ToR boards on here during their open betas...

That sort of behaviour was clearly triggered by those companies themselves that made it very clear people were not in beta for actual testing. People really interested in testing got burned/disappointed, and guess who is left to visit the test servers now: people that only want sneak peaks of what is to come. There are still alot of people out there that actually want to test, but not so much when they are not really needed, but still invited into beta or to live test servers or whatever attracts the sneak peak crowd.

Hmmm,

you could call it alpha instead of beta, have a stiff application/selection process, and make clear that continued participation is contingent upon not only time spent testing but also constructive feedback (choose your medium: email, forums, surveys, bug reports, irc chats, a combination of all of those etc.).

And then enforce that rule.

The whine from people being kicked would also generate free publicity.

I honestly think the majority of testers who want to help test a game (in the old-fashioned way, knowing that things will be borked and not work properly, being willing to repeat action x thirty times for a solid repro, roaming the world looking for spots to glitch or fall through, etc.) would not have an issue with any of that.

They'd especially not have an issue with a dev team that are engaged and responsive with their testers in return. Indie developers usually rock in that respect.

Marketing betas have hurt the testing process terribly, and have hurt the quality of released products even more.

Trouble is, with them as the standard it's hard as heck to find old-school testers wiling to put in the donkey-work to help improve a game. Everyone's a tester nowadays, and the word itself seems to mean different things depending on who you ask.

I guess it's down to companies to outline what testing means with regard to testing their game, and then stick by their own definition.

You actually need to MANAGE your testers.

- Define use and abuse cases

- Track activity of testers

- Support your testers in test methods and improving the quality of bug reports

- Have different testing groups to focus on areas (combat/crafts/pvp/hacking/exploiting/griefing/etc..)

- Evaluate the effectiveness of your testers and replace them where needed.

- Reward your valuable testers.

I think there are many ways to get good use out of your testers, but the key is to keep the group manageable and actually manage them like they were your own employees.

Regarding the load performance test ... i understand their need, but in godsname also call it a load test then.

Regarding the use of beta as marketing tool ... why don't you just release a demo 2 weeks before release  ... that always worked best for me.

When a Preview Tester crashes the game, they curse and rant on the boards, generating bad buzz.

When a Beta Tester crashes the game, they cheer and attempt duplication, possibly making the game better in the process.

 If the Corporate string pullers want the "Beta Test" to be a Preview Test, then they need a new name for the real bug-squashing test that makes it sound unappealing to the average gamer, like the Crash Test, then wait until Crash Test is done, and the only things left are server stress and such. Then announce the "Beta Test".

Well the definition of "beta" has changed a bit.  Beta now means sneak-peek, or pre-order bonuses or at the very least a head start with your character so you can out level the competition.

I'm not going to rant against "The Man" or any of that crap.  Companies are built to make money.

I guess being selective about not only who is invited into initial testing but who to keep.  If you have a tester playing the game for three weeks and they haven't uttered an ounce of constructive criticism then give their spot to some one who will.

Make the testers feel as though they are a part of something.  That this game in some small way is their baby to.  It's their game to.  

It takes a village to raise a child.

Today's beta crowd is vastly different.  Most of them expect a sneak a peak.  Gosh knows when I was in the stress test for swtor all you would read in the forums was folks going I cant log in, and they obviously did not understand the meaning of a stress test. 

When I run in beta I try to break things and send in reports,  but most folks now days expect to play the sneak peak.

I have to say most beta's nowdays are an advertisment at best not an actual beta.  

Being in the firefall beta I suppose  its all about how you view it.   There's no denying  the moment you load the program , that it, thats your first impression.   Its human nature.  now as somone who been beta testing since the late 90s I  have to say  one major mistake I always see.    People think  Beta's are 100% Demo's  Like I said above the moment the game start yes you  will have a first impression  but like all first impressions its not represeantive to the big picture. 

People think  of I'll just silently  play through this beta and  assume they  know whats broken.   That is the WORST way  to be in any gamebeta.   Not only because your not contributing to making the game better, but you have the gall to assume the Dev;s know every little issue.   Developers are ALOT different than players.   They  can get tunnel vision  quickly and lost sight of other things.   Thats why as a player  you are there to remind them of the little things.   A texture missing on a map somewhere.  a gltich  with  mob ai.  All the stuf that would immidiately stand out to you A dev may miss. 

Yes you can have a first impression of a beta.  dont let anyone tell you that you cant.  However DO NOT view beta's  as final Demos.  You are there to help the devs make the game solid  before it even hits the shelves. 

Totally agree with the article.. a Beta test is just that a test. Its there to find as many bugs as possible before they game is released. Also then towards then end of beta testing you usually get at on of people invited for the test to make sure the servers can handle the load..

Over the last year I have tested qutie a few games and usually straight away you get people demanding compensation because they cant play..

For example the recent Tera EU sneak peak had issues from the start. basically straight away you had someone demanding to be compensated..

https://forum.tera-europe.com/showthread.php?t=28873

I dont know why he thought he had the right to anythign but there we go..

Originally posted by Astraeis

That sort of behaviour was clearly triggered by those companies themselves that made it very clear people were not in beta for actual testing. People really interested in testing got burned/disappointed, and guess who is left to visit the test servers now: people that only want sneak peaks of what is to come. There are still alot of people out there that actually want to test, but not so much when they are not really needed, but still invited into beta or to live test servers or whatever attracts the sneak peak crowd.

 

I guess I am old-fashioned then. To me a beta is a test period (and this is not exclusive to the gaming industry). If I am invited into a beta, I am there to test it. It is not exclusively on the shoulders of the devs that this preview tendancy has happened.

If the devs only treat their betas like a trial preview, then it tells me their quality department has serious flaws and it is best not to end up investing in the final product because they do not employ the industry standard of testing whatever they release.