Jon Wood: Grinds My Gears: Critique Our Reviews

I’ve taken a lot of shots over the last few weeks at developers and players alike. So now it’s time to set my sights internally and take a look at the journalism side of things, specifically where reviews are concerned.

For a long time, I’ve been dissatisfied with the ways that MMORPGs are reviewed. Most of the time, they’re reviewed way too early, without much or any consideration given to the long term nature of the genre, journalists are playing the beta and calling it a review, the current scoring system is broken and more.

So we’ll start at the top. The reason that games are reviewed too early in many cases is simple: Gaming websites are in a position where we have to try to get a review of a game out the door as close to launch as possible because that’s what’s demanded of us so that you all can make a more informed choice as to where you put your money.

The problem, as I see it, is that We’re talking about two separate issues when we’re talking about MMORPG reviews. One requires swift and immediate feedback and the other requires laying low for a time and experiencing more of the game.

MMORPG reviews are beholden to two masters: First, whether or not the game is worth the initial box price and 30 days that come with it. Second, whether or not the game is worth the long term investment of an ongoing subscription fee.

I honestly don’t know if one, single review can actually serve both of those masters in a realistic way. A reviewer working within the time confines of answering the first question has no way of answering the second. As a result, readers have n idea what the review is pointing at or what it’s trying to tell them and confusion abounds.

We’ve tried to solve that problem here with a launch day “preview” of the game and then a later follow-up review, but there still seems to be a good deal of confusion happening and people have expressed a dislike of our preview system as well. So, here’s your chance. I’ll admit full well that the way we’re doing things isn’t ideal, and that change is needed. I’ll also admit that you guys probably have some solutions brewing in your heads that I haven’t come up with so I invite you to share them with me in the comments. The editorial staff will be watching very closely.

Next, I wanted to talk about scoring systems. over time, I’ve become more and more annoyed with the 1-10 scoring system. The trouble is that numbers 1-5 are very rarely used because they represent a failing grade. For me to give any game a 5 or below means that it probably didn’t even run on my machine or it so wildly missed the mark that it doesn’t even resemble a running game anymore. So why, if it’s never or rarely used, are we taking up half of our scale to say the same thing: This game totally fails.

Then there are the degrees that a 1-10 scale creates. Who, other than the reviewer, really knows the difference between a 6.5 and a 6.6? How does that difference stay consistent from reviewer to reviewer? Heck, how does the scoring system stay consistent from reviewer to reviewer anyway?

I know that 1-10 is how most gaming sites do things, and many of them do it because it puts the score into a nice little package for MetaCritic. So I’m wondering, where MMORPG.com is concerned, how important is it that we stay on par with everyone else? Do you all enjoy comparing the numbers from one site to the next? Is that helpful for you as a consumer?

If not, and you think we should change it. How should we change it? I’ve always liked the idea of using a letter grade system. A+ to F. At least then only one portion of the scale is relegated to failure, and everyone seems to have a clear idea of what the system means.

Then, there’s the idea that maybe we should go back to our roots and rate a game by 8-10 categories and use an average of each of those to get a score. That’s how MMORPG.com used to rate games.  It solves the confusion issue, but should something like sound be held on even par as a category with gameplay or innovation?

Or maybe the solution is something completely different, something that you guys have seen and liked or even dreamed up yourselves.

As the Managing Editor of this site, I recognize that we could be doing things better. Please take the time to give us your constructive feedback so that we can change in a positive way.