Reviews are the hardest thing for any site to do. It's a lose/lose situation. No matter what the score, someone is always going to be upset, so generally, our policy is to give the author of the review the clearest guidelines we can, then stick with what they say. They, after all, played the game with a review in mind and as much as we might like, we cannot all play every game. We have to trust the reviewer.
Right now, we use a scoring system out of 10. Generally, we tell our reviewers that they should look at it academically. In school, generally speaking, the scale looks something like this:
- 10 = A+ (perfect)
- 9 to 9.9 = A (great)
- 8 to 8.9 = B (above average)
- 7 to 7.9 = C (average)
- 6 to 6.9 = D (below average, but not unacceptable)
- 5 to 5.9 = F+ (failed, but not totally without redeeming qualities)
- 2 to 4.9 = F (failure, with lessening degrees of redeeming qualities)
- 0 to 1.9 = F (totally without merit)
That's the logic we use in our scores, but I freely admit it is not a perfect system. No system that applies scores to games, let alone games that continue to evolve and change like MMOs is going to be totally fair.
Fact is, no matter how hard our reviewer tries, there are only so many hours in a day and they cannot possibly do everything in a game. We can only pay them so much, and reviewing an MMO is a temporary full time job.
Editorially, we insist on at least three weeks with a game before we review it. That is longer than most sites, I believe (how the heck do some sites have a review ready on launch day?) and gives the game a fair shake.
We are also going to re-review games far more frequently moving forward. Whenever enough time or major updates have come by, or even if a company comes to us and says "Hey, it's much better, want to give it another try?" We're happy to give games second, third and fourth looks. These are not console games, they change and we'll constantly go back to try and bring it up to date.
Nonetheless, there will be imperfections. Obviously, the Darkfall review was bound to be one of our most controversial. Fact is, there is absolutely no right answer there. Every single possible number was going to piss off a third of our community. So, we did what we always do and trusted the reviewer.
So, I throw this out for everyone to chew on. How would you score games on MMORPG.com? Keep in mind the following rules:
- It must be a numeric score. Not everyone uses things like letter grades, numeric scores are more standard.
- It must be the same for our hype/rating meter as it is for the review itself.
Beyond that, we'd love to hear your feedback. We always want to improve and since the reviews are done to help people make informed purchasing decisions, I figure it's best to ask you all how you'd be best served by them.
Feedback away in the comment thread!