Justin Webb: Two Thumbs Way Down

As most of you probably know, Metacritic is an aggregate review site. It takes reviews from "reputable" sites, converts those reviews into percentage scores, and then averages them, resulting in a "metacritic score". Metacritic can be a useful tool if you want information regarding a new game. What's the "word on the street"? Is the general consensus good or bad? When a new-release game costs sixty bucks, this kind of consumer information is really useful to have at your fingertips. Who wants to spend $60 on a crap game?

Given that a Metacritic score is an average of review scores from multiple popular gaming sites, it sounds like it should be pretty accurate, right? Well, here are the Metacritic scores (as of 2/16/2010, in descending order) for some "recent" and "big-name" MMORPGs:

I'll let these scores speak for themselves. ... Actually, no I won't. Clearly, there are some extremely ridiculous scores on this chart, scores that are making me physically angry just looking at them. I'm not going to editorialize over these scores, or point fingers, but I encourage you to in the forums.

So, how come these scores are so messed up?

There's a bunch of reasons, many of which are unique to MMOs and how MMOs are reviewed; some of which are inherent to how Metacritic "works"; and all of which can be abused and manipulated.

It should be noted that the scores above are, by and large, launch scores. Many MMOs have very polished and pretty starter areas. How far did the reviewer get in the game before committing pen to paper? Also, some companies have perfected the art of marketing and hyping an MMO. In combination, smart marketing and an awesome demo can easily nudge a review a few points better than it should be, especially if the reviewer isn't prepared to invest loads of time into the review.

Many "small" review sites are under pressure to have "exclusive reviews" - i.e. publish their review first. Obviously, the less time you spend playing, the more exclusive your review can be. And, Metacritic doesn't care if your review site is "small" -- your review is assumed to be as legitimate as one of the larger game sites, and given as much "weight".

It's notoriously difficult to review an MMO. Many games only really start to kick in during their end game; or, conversely, have terrible end games - a part of the game that no reviewer is going to reach while writing their review. Some sites have recently implemented policies whereby they will not publish a review of an MMO until a month after launch, giving their reviewers ample time to come to a decent decision on a game. This is a great first step at ensuring that MMO scores are real and accurate, and for maintaining the integrity of a site, but can be a double-edged sword for consumers. Where do you get information when a game releases? The site under a self-imposed embargo (no), or the site with the "exclusive"?

Also, you only get one shot at launching a game. If a released game is clearly not ready for release, it's (historically) really difficult to come back from that (critically), regardless of how good a job you do at fixing stuff. Very few sites ever go back and re-review a game later. And if they do, it's uncertain whether any revised score would be picked up by Metacritic after the fact. As an example, we recently voted Age of Conan as our Most Improved MMO. However, Conan's Metacritic score will remain as a launch snapshot of 80 regardless of how much better it has become since then. There are plenty of games out there that were mediocre at launch (and consequently got lousy Metacritic scores), but that got whipped into shape by their Live teams later.

I should note that I actually quite like Metacritic. I use it a lot, especially if there are lots of games that I want to buy but can only afford one. I use it as a "guide". For console-game purchases, it's a really useful tool. However, I've come to realize that the aggregate review process fails badly when applied to MMOs, and those scores should be treated with the utmost suspicion.

So clearly, Metacritic scores, especially MMORPG ones, are wildly inaccurate and should be taken with a grain of salt, right? No-one actually uses this data for anything important do they?

Errmm, actually Metacritic data is used a lot by game publishers. At some point in the last few years, the big publishers decided that Metacritic was the most awesome thing evah. Imagine the scene, a boardroom full of suits realizing that an internet site exists that can empirically tell them how good each of their products are. All of a sudden, performance reviews become much easier. Studios can be ranked against one another based on their aggregate "scores". Metacritic become an internal assessment tool. Suddenly, a publisher's "worst" studio could be identified. It literally makes me want to puke if I think about whether Metacritic scores were used to determine who got laid off last year.

Somehow, Metacritic became the quality yardstick for the video-game industry. You might think I'm exaggerating, but Metacritic scores are a huge HUGE deal behind-the-scenes in the industry. The problem is, Metacritic is broken when it comes to MMOs. It just can't do them right (as evidenced by the above score chart). Many companies treat Metacritic scores as gospel, and put as much credence in them as straight up box sales. I've been at one company where a chunk of my bonus was directly proportional to the Metacritic scores of the projects I worked on.

Take a closer look at some of the investor reports that were released last week. Metacritic scores, and their inferred fiscal importance, are sprinkled liberally throughout all of them. For example, EA loves Metacritic scores and is always harping on about them (80 is their "magic number").

One of the main problems is that MMOs don't really lend themselves to being reviewed. The experience is perceptual, personal, and subjective. But Metacritic goes one further. It takes nebulous concepts such as "good" and/or "B+" and converts them into a percentage score, making value-associations where none are there.

We can't have it both ways. If Metacritic is going to generate bogus scores for MMOs, it can't also be used as a financial tool to determine whether developers are successful or not. That's just not fair.

To the forums!