MMORPG.com columnist Justin Webb writes this article about Metacritic, a great resource for single player games that doesn't necessarily take into account the intricacies of MMORPG reviewing.

As most of you probably know, Metacritic is an aggregate review site. It takes reviews from "reputable" sites, converts those reviews into percentage scores, and then averages them, resulting in a "metacritic score". Metacritic can be a useful tool if you want information regarding a new game. What's the "word on the street"? Is the general consensus good or bad? When a new-release game costs sixty bucks, this kind of consumer information is really useful to have at your fingertips. Who wants to spend $60 on a crap game?

Given that a Metacritic score is an average of review scores from multiple popular gaming sites, it sounds like it should be pretty accurate, right? Well, here are the Metacritic scores (as of 2/16/2010, in descending order) for some "recent" and "big-name" MMORPGs:

Read Two Thumbs Way Down.

STO higher than UO??

I need to leave this planet ASAP.

Hi Justin

Quote: "So, how come these scores are so messed up?"

Maybe you could elaborate whats so wrong with those scores? Looks good to me. Hard to understand your article if I can't see the your point of view.

Edit: Newer Games (see STO) might move up a bit more in Metacritic as more Reviews are released but thats about the only one that seems a bit ona low the rest looks very much like mainstream consensus.

I dont really buy alot of games tbh, I am very well acquainted with my tastes, and usually know before hand if Im going to like a game or not. And Im usually right. So reviews are useless to me. I read them after I play games though, to see what people thought about the things I experienced in-game.

Hello,  just something worth taking a look at is the way as Metacritic uses the reviews here

www.metacritic.com/about/scoring.shtml

They are weighted averages and we definitely do not now how weighted they are.

Moreover lots of  the linked reviews are not accessible anymore.

I agree with the final opinion that for some games (and especially mmos) the resulting score is not appropriate, but for 90% of them I usually agree with the score they give.

I suppose it also depend on the state when mmos are delivered which is usually a beta, and the reviewers gives out their score for that as well.

This last is definitely the reason behind the whole thing.

Cheers.

Originally posted by Tykey

Hi Justin

Quote: "So, how come these scores are so messed up?"

Maybe you could elaborate whats so wrong with those scores? Looks good to me. Hard to understand your article if I can't see the your point of view.

Edit: Newer Games (see STO) might move up a bit more in Metacritic as more Reviews are released but thats about the only one that seems a bit ona low the rest looks very much like mainstream consensus.

I agree. Those scores don't seem outrageously inaccurate. Maybe WoW isn't that great of a game, but a lot of people seem to like it and play it. And maybe STO is a good game, but it's getting a lot of hate in forums and reviews because it's not MMO enough, you can never satisfy the hard-core fans of an IP, and it's not cheap enough for some people.

 With youtube vids, beta breaches, copies being released before "official" release dates, and general word of mouth, who the hell needs Metacritic or any other review site nowadays?

Well first, before saying the scoring system is flawed and innaccurate you should look at yourself first cause your using the same system on this site.  Game score is an avrage of all the scores, maybe that's why you have a game like Atlantica Online being rated #1.

However i dont think a "serious" review site would give an MMO all 10's or all 1's just to try to cheat like we have here from users making multiples accounts just to bump or lower a game rating.

In ALL a system that makes an average of all the scores seems accurate to me, as long as the reviewers are somehow serious in their jobs.  As far as the scores in this article, they seems FINE to me.   Of course WOW is number #1, but WOW is like McDonalds.

---

Originally posted by Uskyrt
They are weighted averages and we definitely do not now how weighted they are.
Moreover lots of  the linked reviews are not accessible anymore.

---

Anyway, that shows a little how weighted the score are. We can imagine how one score on IGN can bump up or down a game'S score. Yes, the more reviews Metacritic aggregates (in my example there are only four), the closer to "reality" the score should be, as excessive enthusiasm should be pondered by excessive negativism, and vice versa.

And yet, we still got skewed scores.

Honestly I'd never heard of this site before and now that I know about it I doubt I'll bother with it for the reasons you described in this article.

I'd like to say that I'm disappointed that developers would take a number from one site and base all their praise (or lack there of) on it. I'd like to think that developers have people who go out an research what people actually say about their products and look at things beyond just a made up score. But unfortunately I know better.

Nice read.

When you say MMORPG's are not easily reviewed, I think you really hit the nail on the head. One of the X factors that is (almost) never mentioned when reviewing an MMORPG, is the community. Community is generally something that's beyond the marketing departments mental grasp, and therefore beyond their control, yet is crucial to a games success or failure.

Bit of a tangent, but it's something I've been thinking about, and I think it's somewhat related to your article.

In my opinion MMOs should be reviewed yearly, and the older reviews and average still displayed, but remain for year X.

So MMOs should have:

1- The aggregate score from all years and that is what is displayed for users on the search, the main score.

2- The aggregate score of each year and their review listings.

3- And a system in place to automatically make these changes as each MMO completes an year of existence so that reviewers can notice the missing reviews for the year.

Now, good luck convincing Metacritic and the industry that MMOs are software that isn't static and get content patches for free frequently (it's a damn evolving world) including DRASTIC gameplay changes... completely opposite to single-players where content patches will be usually either DLC or Expansions. Shouldn't really be hard to apply when they understand this aspect of MMOs, this would be already a great step into giving MMOs some justice in reviews.

Anyway, I do use Metacritic for weighting some game purchases, especially console ones.

Well i say mmorpg.com over years ain't good reviewers also so you can hold hand with metacritics hehe.

But i agree on mmorpg been reviewed get scores when launched and it dont chance anymore harm done, there are plenty mmo's out there that are so much better after 6 months but never got a chance to recover from those bad reviews even some died while there very good after 1 year or more.

I dont buy games becouse of reviews i investigate myself follow i game i like and with info developer, forums and players who play beta i deside if its worth or not if i think is risk buying at launch i just wait for while and see if its more polished.

So no mis buy on my part.

Originally posted by EricDanie

In my opinion MMOs should be reviewed yearly, and the older reviews and average still displayed, but remain for year X.

I heartily agree with this, as MMO development is generally an evolutionary process.

Metacritic gives you a number and points you to all of the reviews that go into said number.  It's up to you to determine how useful that number is.

Good post!

You can definitely see a change of moods and standards in those scores.  I think the low UO score comes largely from having little to compare it to, for example.  I think reviewers likely compared it to the Ultima series and thought, "wow, it's like other Ultima games except without the purpose.  Oh, and getting PK'ed all the time."

2003-2004 roll along, and it seems that no one could do any wrong.  AoC would come along a few years later with the bait and switch that is Tortage, which totally skews the game perception to be something that it isn't.  This would give it undeservedly high reviews for all the wrong reasons.

The most interesting to me is WAR.  I recall the consensus being that it was an attempt to be WoW 2.0 with a deeper pvp focus and that they had pulled that off to some degree.  But the devil was in the details, or the lack thereof, and now it seems more like WoW .3 with a few interesting innovations.

It is tough to review MMO's, I agree with Justin.  To make a fair assessment, you'd have to play for months.  The only REASON you'd do this as a reviewer is if you enjoy the game.  So then, only "fair" assessments would be done by people that enjoy the game, thus skewing the chart even worse.  And that's not even talking about the logistics of being a paid reviewer getting a few hundred bucks for 100's of hours of your time.

Originally posted by Toquio3

I dont really buy alot of games tbh, I am very well acquainted with my tastes, and usually know before hand if Im going to like a game or not. And Im usually right. So reviews are useless to me. I read them after I play games though, to see what people thought about the things I experienced in-game.

This hits the nail on the head for me and pretty much sums up my feelings and input.
 

Originally posted by TookyG

Metacritic gives you a number and points you to all of the reviews that go into said number.  It's up to you to determine how useful that number is.

Yes but that's what he's saying.  That number can be excessive or correct; but there's no way for us as consumers to know how many numbers went into creating that number (the number of reviewers) and there is no way to verify how accurate the score is versus real world play, and finally that this number is totally static and won't be dynamically altered to reflect patches and new content.

Plus to add insult to injury the CEO's of all these MMOG's treat the damn number as if it's from God himself; and also it's a statistic of a statistic which anyone who knows statistics knows is a BS number!

This is the Harry Mudd dilemma.  "Everything Harry tells you is a lie!  Listen carefully, Norman, I am lying!"  You can't tell which scores are accurate or not.  The only thing to do is to ignore the whole lot of them!

Originally posted by MrcdesOwnr

Originally posted by Toquio3

I dont really buy alot of games tbh, I am very well acquainted with my tastes, and usually know before hand if Im going to like a game or not. And Im usually right. So reviews are useless to me. I read them after I play games though, to see what people thought about the things I experienced in-game.

This hits the nail on the head for me and pretty much sums up my feelings and input.
 

I'll never buy a MMO without a solid 20 hours to play it beforehand.  That's usually enough time for me to know if it will at least amuse me for a short time.

General scores mean little to me, I agree there.  But specific details about game mechanics are important to me in a review.  When reviewers bring up how the "crafting" works in STO, for example, I think that's an extremely important and invaluable piece of information.

But yeah, general, average percentile scores mean very little to me.  A game can get review scores based largely on a  mode of gameplay I may have 0 interest in in the first place.

Sites like Metacritic and Gamerankings.com suck.

If you follow those scores like a sheep you're part of the problem.