Jamie Skelton's recent column got me thinking. Until recently, beta access was something offered by invitation only and usually only to those the dev team decided they wanted there. Then came the public betas, offered as incentives to purchase a game pre-release or simply as a free trial to give the gaming community a taste and hopefully generate some buzz for their product.
As many have noted, beta used to be for serious testing and pre-release bug killing. These days, it's become more of a promotional tool than anything else. Ms. Skelton defines the problem well but she doesn't offer a solution. I'd like to make an attempt at one here.
The main question to answer here is a simple one: What is the best and fairest way to distribute beta access invites? What method offers the greatest benefit to all involved and offers good incentives to participate for both developers and players?
How about making players earn the right to be invited?
Many games utilize ranking/killboards to track player goals. What I propose is a website that ranks beta testers in much the same way.
Players who are in beta tests for various games would track their bug reports through this site, in much the same way as EvE players track their killmails. Players would provide demographic information that would interest dev teams such as physical location, PC hardware, connection speed, etc. that would be accessible to the devs through the site,
While players would provide detailed bug reports and demographic information to the devs, other players would only see basic information such as number of reports filed, in what game, and a player's overall ranking. For their part in the process, the devs would need only to rank the reports by some basic categories such as whether a report is an actual bug or not and its severity, and then resubmit them to the site.
Points would be awarded based on the dev rankings and players would be ranked not only on success but also on effort (i.e. credible reports filed, even if they turn out not to be actual bugs). In other words, the point system would be structured so that you don't necessarily have to be a master bug hunter to have a high ranking, but being one certainly helps, a lot.
When the dev teams are looking to fill beta slots, applications would have a place to submit a player's ranking from this site. Players with higher rankings can expect it to be more likely (though never a guarantee) that they'll be higher than most on the invite list.
The way I see it, it's a benefit to the players because it offers something to compete for, a way to earn the right to be on the "A-List". It's a benefit for the devs because they can pick and choose among applicants with proven records of bug hunting, as well as the ability to easily track who's in their beta, where they're playing from, what hardware they're playing on, and possibly a lot more.
Another benefit for the dev teams is that it helps encourage more beta testers to actually beta test rather than just play the game because this method would make beta testing a metagame in itself.
Obviously, there would be a lot of details to be worked out. Basing the rankings upon the dev ratings rather than just the number of reports filed would help to keep players honest. Just as reports found to be valid/credible would gain a player points and thus a rise in their ranking, too many invalid reports would cost a player significantly.
Once this system has been in place for a while, the benefit to the dev teams will likely increase even more. Imagine a year or two after this system is in place that a dev team wants to put together a beta test team, and they've got, say, 200 free slots left after all the necessary invites are done. This dev team could go to the site, and pick and choose its beta team by success rate, specialty (the type of bugs a tester is good at finding), which games/betas they've been in before, and/or any of the many other statistics they'd have available to them because they'd all be tracked through the site.
New players who want to become involved and become ranked on the site would still have the open beta periods during which to earn points, but invitation to closed betas (and perhaps even alphas) would be offered to those who the dev team believes will best help them accomplish their goals and make their game the best it can be.
To my way of thinking, it's a win-win. Beta testers gain something valuable to compete against each other for, and devs gain reams of valuable demographic information on interested beta testers, and more reliable and consistent bug hunting from their beta teams because they can be chosen by skill and reliability to help address specific issues.
Presumably, the end result is that everyone wins with fewer bugs surviving past launch in new MMO's.
Frankly, I'm amazed that no one (I know of) has thought of this before now. How do you motivate gamers? Make it a game, with winners, losers, and a fair battlefield on which to compete.
So, have I stumbled onto something promising here or have I lost my mind in thinking this could ever work? I honestly don't know. That said, I also don't see any reason why it couldn't work.