For MMORPGS I feel that you should still post two reviews. At first you should factor in that we do/do not believe this game is worth the initial purchase. Then after you have had time to play the game, you give your opinion on whether this game is worth a longer-term investment. 

As for the reviewing system. I believe that you should review the several different parts of the game, splitting the review up. As this is all opinionated, I believe that whoever creates the review should give ratings to all the seperate part. Then the total grade that they believe the game deserves. It should not neccesarily be an average. As you pointed out, while everything that is in the game matters. Some parts just matter more. 

Therefore, the major factors of a game should all get graded seperatly, then as a whole you should decide what the game deserves as a rating. WITHOUT looking at the average grade.

Originally posted by maskedweasel

Originally posted by Moirae

 

Just as an example, STO. Most of the people with more than a years experience in more than one MMO would call that game garbage, even now after its been out for years. Yet the reviewer on here was talking about how truly wonderful the game had become. I mean seriously... that review should never have been released. 

 

Every game in the last 5 years has just been a rehash of other games that came before them with a few changes to make them seem novel, but then they're released and you see how bad they are.

 

And here is where you fell of the wagon. Regardless of a rehash of mechanics thats not to say that games can't still be well made, and enjoyable.  If you really look at gaming as a whole,  all genres are mostly basic copies of the games before it.  

 

Sure a platformer may have changed from 2D to 3D but does that really change the focus of them entirely?  As the MMO genre unfolds we'll start to see more hybridization of styles,  its inevitable.  Just because one game is using a common system doesn't mean that the game is suddenly garbage.

 

Too many people are stuck in their MMO hovels scoffing at games that can't meet their standards and thinking everything that isn't appealing to them is beneath them.   While I didn't really get into STO,  nor did I play Champions for an extended period of time,  I would be an idiot if I though that either game was complete garbage.  Both games are quite playable,  they each have very subtle nuances and changes in what makes them interesting,  and while they aren't worth a 9.5 on the gaming scale,  they definitely aren't worth a 6 either.  (difference between an A and a D for those who love letter grades)

 

"Playable" is fine, but that doesn't mean they're good.  Pretending they are is doing them and the industry a disservice. Accepting "mediocre" as good is a real problem. Its like American Idol. They keep letting the mediocre artists win when the good artists are the ones they vote off. 

Originally posted by bunnyhopper

Whilst I am in no way advocating the ideas of the poster you are responding to or any claims of what are tantamount to 'bribes'. But do you really feel the underlined demonstrates the professionalism you are talking about in the centre of your post, the crux of your argument?

Sometimes people need to be told plainly that they are wrong.  Sometimes you get tired of repeating the same thing and need to drive that point through people.  It isn't the first time people have accused the site of taking bribes for good reviews.  None of these staff are under any kind of contractual obligation to be nice to anyone here.  The header of the site does not say, "MMORPG.com: a place where we are nice to you and also review games".  Does it show a little unprofessionalism?  Sure.  This community is pretty notorious for baiting people into flaming them and then crying that people are being mean to them until that person it banned, so maybe a little unprofessionalism is what is needed.

One thing I would say though is the sites reviewers should keep in mind they are reviewing mmos (often with subscriptions) and not stand alone, offline games. Whilst that may seem blindingly obvious, I have noticed mmos with little content and little real potential (i.e. themeparks with no scope and not enough content) still getting good scores. An mmo needds to have content, or it needs to have masses of potential for player made content (sandbox), if this isn't the case it needs to be hammered home in your reviews.

 

While I don't disagree that maybe a distinction needs to be made between games with a genuine ability to allow players to create their own content and a game that does not, subtacting points based on that is a bit much.  Last I checked the most populated games out there were themeparks, which implies that the themepark style, regardless of what the vocal minority here in this forum would love everyone to think, is still a fan favorite.

Frankly, not everyone wants to be Uncle Owen, and get incinerated by some griefer's flamethrower, as happened in the actual movie to Uncle Owen.

The simple solution to the rating dilemma, in my mind anyway, is to clearly state what the points are for.  Allot points based the factors of the game.

 A set amount for sound quality and diversity.  

A set amount for the graphics without giving a handicap because the graphics are, "stylized", which is code for "I didn't want to do a lot of work so I made the characters look like steroid abusers".  

A set amount for control fluidity, and so on.

Letting people reading your reviews see exactly what these points are being based on instead of, "I feel like.. 6.8 today.  6.8 it is." would probably go a long way.  If something like that is already in place, maybe have it linked next to review scores as a kind of, "HEY THIS IS WHAT THESE NUMBERS MEAN" type of thing.

Originally posted by Moirae

Originally posted by maskedweasel

Originally posted by Moirae

 

Just as an example, STO. Most of the people with more than a years experience in more than one MMO would call that game garbage, even now after its been out for years. Yet the reviewer on here was talking about how truly wonderful the game had become. I mean seriously... that review should never have been released. 

 

Every game in the last 5 years has just been a rehash of other games that came before them with a few changes to make them seem novel, but then they're released and you see how bad they are.

 

And here is where you fell of the wagon. Regardless of a rehash of mechanics thats not to say that games can't still be well made, and enjoyable.  If you really look at gaming as a whole,  all genres are mostly basic copies of the games before it.  

 

Sure a platformer may have changed from 2D to 3D but does that really change the focus of them entirely?  As the MMO genre unfolds we'll start to see more hybridization of styles,  its inevitable.  Just because one game is using a common system doesn't mean that the game is suddenly garbage.

 

Too many people are stuck in their MMO hovels scoffing at games that can't meet their standards and thinking everything that isn't appealing to them is beneath them.   While I didn't really get into STO,  nor did I play Champions for an extended period of time,  I would be an idiot if I though that either game was complete garbage.  Both games are quite playable,  they each have very subtle nuances and changes in what makes them interesting,  and while they aren't worth a 9.5 on the gaming scale,  they definitely aren't worth a 6 either.  (difference between an A and a D for those who love letter grades)

 

"Playable" is fine, but that doesn't mean they're good.  Pretending they are is doing them and the industry a disservice. Accepting "mediocre" as good is a real problem. Its like American Idol. They keep letting the mediocre artists win when the good artists are the ones they vote off. 

Well, "good" is subjective.  While I didn't like Cryptics newest games enough to subscribe to them,  I do think they are worth a play through.  Mediocre games are just that, some gamers like games that are more of a diversion while they wait for something else.

On the other hand some people may just really dig being in the Star Trek Universe,  and while some people will likely complain its not Star Trekky enough,  the game is completely playable,  has some mechanics that are interesting.  Every game doesn't have to knock everybody out of the park.  If some games can just hit their core demographic well enough then I think they'll do fine.  More people are expecting every game to stand up to World of Warcraft.  What it comes down to though,  it really doesn't look like Star Trek Online was trying to do anything but cater to Trekkies in the usual Cryptic Fashion.  They aren't bad developers,  they just don't shoot for the moon either.

Look at the "best in class" publications and critics.  The common way of doing things is to have a preview (a.k.a. "fluff article") where someone that's anticipating the game writes up what it's all about.   This part isn't about criticism.  It's about passing on whatever information you deem important to the reader.  

It is quite often that I will read on a thursday a detailed story about how a "brilliant" director with a "stellar" cast are working "on an original vision" from an "amazing" screenplay.  Then on friday, when then film actually comes out, I'll read a one-star review telling me about the totally flat direction, terrible acting and laughable dialogue of this same picture.  

There are two different jobs.  Previewing and creating interest and excitement are one thing.   Criticism is a different thing.  A "Review" is about criticism.  

As far as rating systems...honestly, it's all the eye of the beholder.  The best film reviews I read are in the New Yorker.  There is no rating of any kind assigned to them but it's abuntantly clear what the reviewer thinks and whether something's worth it.   So the score doesn't matter, it's what the reviewer has to say.

That being said, if you're going to introduce a new scoring system, I think it's important to communicate what every rating means.  Whether that means having a permanent legend or having the reviewer say: "i'm giving this a 4/5 because it's worth playing for a long time, but is still not 100% stable".  

Given that there are so many aspects to MMORPGs and that every fan values them differently, it may be wise to use a set of ratings rather than just one.  Complex things are often ranked on a set of attributes.

What makes up a MMORPG?

Setting & Lore

Solo Content

Group Content

Endgame Content

Crarfting Content

PvP Content

Immersion

World Interaction

Graphics (Technical)

Graphics (Artistic)

Stability / Bugginess

Customer Service

Promise (what's coming)

Value (Price/etc)

That's just stuff off the top of my head.  But that's stuff that can be defined and outlined somewhere for your reviewers to use.  What is the gold standard in all of these?   Decide and state.

No matter what you do, reviews are going to be subjective.  The key for a fan is to find a reviewer whose opinion they respect.   There are 4 daily newspapers in my city that all have film reviews.  I know exactly what i think of all the reviewers in those papers, so I know whose review I'm going to pay attention to.  Your people don't have to be 100% consistent with each other.  They are individuals after all.  At the same time, there aren't that many MMOs out there.  Get multiple people to review major releases.  

Those are my thoughts, I have one last thing to add, but it's kinda separate from this.

~~~~~~~~~~~~~~~~

Why is it that MMORPG journalists NEVER ask any tough questions of developers or press developers for real responses when they're being brushed off?  Getting to the real story is part of what a journalist does.  

If you ask a SWTOR developer "How can you possibly design a major part of an MMO to be single-player only?"  and the response they give you is "We love star wars as much as you" - that is complete and utter bullshit and any self-respecting journalist would be insulted by the answer.    

For the love of god, PUSH!     And if you don't get an answer assume the worst and write it, so that next time they know not to feed you bs.   You are not idiots.   People are not idiots.   It often times seems that the writers of reviews and previews are the only people that are incapable of reading between the lines of what's being said.

I think the reviews would benefit from being scored relating to different gamer types. For example:

* Achievement hunter

* Trader/hoarder of stuff

* Social gamer/Guildie

* PvP/PvE gamer

* Explorer

It would be nice to read your opinion about how the game fits different gamer types either during the course of the review or in a review summary. I think this would help relate to the game being reviewed.

For me, I normally glance at the pro and cons reviewers give to see if any (personally) game-breaking cons are listed.  Then I look to numbers only for technical or semi-technical ratings like performance.  Finally, I don't care what number a reviewer gives to gameplay or fun or things of that sort.  I just read what they write.  Lastly, I look at the overall grade for a reality check.  Bottom line for me is whether or not the pro v con list scares me off. 

Good article.  On a side note, the whole concept of journalistic integrity I find silly.  It's like people want to legislate and regulate deeply personal decisions an individual reviewer makes.  Give me a break.  It's impossible to know what influences a person really has without really getting ot kow them.  Even then it's just conjecture.  Take their review for what it is and make your own choice.

Finally, happy Friday everyone!  Have a great weekend! 

I think when trying to decide on a new format you need to think about how that format would apply to existing games.  I'm not sure one format is going to fit all - for example:  Darkfall, WoW, and Free Realms all are MMOs.  The folks looking for a new game are going to want to know if they are good games for them.  So target audience is going to have to be identified and any grades/rating reflect that fact.  Ideally a member of the target audience would be the one doing the reviewing.

What would be nice is to know how each of the following grade a game:  raider, crafter, pvper and soloer.  I expect there are some reviewers that can fulfill multiple roles but even then we're talking a lot of reviews because MMOs evolve and what was experienced on day one will be totally different within a few months.

Ultimately if I know the reviewers game background and if they themselves are willing to shell out bucks for the game that is more information than any score could ever give me.

The biggest thing that is needed with your scoring system is some kind of consistency in the ratings.  I remember at one point a review for Age of Conan gave it a very low score (like a 6 maybe, I didn't go look it up).  That was probably one of the lowest reviews ever.  To put it in comparison, Vanguard scored better.  So did Matrix.  So did SWG for the NGE review!

That really ticked me off.  Sure AoC had issues, but worse than the NGE?  Worse than Vanguard?  Seriously?  I do not know if the reviewer was a big PvPer (as so many at MMORPG.com are.  Nothing made me chuckle more than early reviews of DCUO that started aout by talking about the issues like bad chat but then got all giddy as a school boy talking about how great the open world PvP was and that made the game AWESOME....anyway) so maybe he was just upset that all of his Sieging dreams did not come true.

Regardless, scores should be meaningful.  if you give something a 4, any game that gets a 4 needs to be in that same category.  Maybe the final score should be tweaked by the Editor to make sure that there is consistency.

Originally posted by ghstwolf

A breakdown by category would be my prefered answer, with 2 planned reviews.

A technical biased "early look"- heavily concerned with stability and performance matters.  Really it's a what is in the game and does it work style of review.  It would be nice if you could also do installs on high and low end rigs, playing around enough to judge the differences.

The second review- the resub review a month (roughly) later.  This would be the more "traditional" review, grading how well things work together.  By this time the community should have settled a bit (and its importance gauged), you are into the typically boring middle game, and you'd have had time to play around with the systems.  This is also a good chance to "update" on any technical improvements.

Such a system would be a huge improvement.  Placing the review focus on if a working product was delivered early on, then assessing its lasting power with the review a month in.  It probably won't make many friends though.  Devs will dislike the requirements on content/gameplay, since they won't be able to hide the reality of their game behind the initial "hook" hours.

 

I agree with this. Then you can also add a final score so you have something for everyone. More options is always nice imo. Although, personally I don't have major problems with the current system also, so..

I think arieste just became my personal hero. :D Talk about hitting the nail on the head. 

Its like when EQ2 went F2P and we all started asking for direct answers to direct questions in detail on how it would work and what it would do and we kept getting blown off then got upset and were totally ignored.

PUSH. Get the answer. You're reporters, so do your jobs, be fearless, and get it. 

Lemme keep it simple...

Return to your roots.

Score a game on several key "MMO" specific features and toss in a bonus (x2) score for "innovation" or "creativity". Total up the 1-10 scores, divide by the number of catagories and be done with it.

The most important thing is establishing specific items you feel are key to the MMO genre you represent (grouping, guilds, chat etc). Any game missing one of the pillars of traditional MMOs will get a 0 in that catagory (housing for instance), but they would be able to make up for it with something "innovative" (Rifts, action combat, voice acting).

Heck with reviews, I would be happy for some SOE outtage coverage here as of late. If armchair analysts have been putting up a slew of serious updates and articles - I am most eager to see what actual analysist without weak wills and with integrity can do. Anxiously awaiting those news pieces. Especially since they do have contacts with developers and can give us honest feedback.

In any case, this isn't the thread to talk about whether or not you think our reviews are biased, etc. it's a place to talk about the direction that the format of our reviews should take in the future. We could continue arguing, but it's just going to boil down to the same stuff it always does. People are going to make accusations and I'm going to respond to them and no one will be more convinced of anything either way. 

If people want to think we're corrupt, then that's their perogative, I suppose. 

So, if I overreacted,then I apologize. I'm tired of having to defend myself from accusations against my integrity on a daily basis and I lashed out. 

In any case, let's get this back on topic please.

I dislike the overall number for a MMO. No matter if its numbers or letters. As long as there is no consistency between different reviewers, then that score is useless. You guys have to remember that the readers only compare scores and not include 'oh but thats reviewer x  and he gives relatively high scores compared to reviewer y' etc.

I do like a pros/cons list though. But instead of just naming some random features or whatever the reviewer thinks of, why not make this list consistent? Make a list of points to which you put the MMO against. The reviewers personal view can then be put in his conclusion at the end of the review and maybe with added comments if the pro/con list isnt complete enough.  Maybe its an idea to hold a poll for which things should be on the list.

Then there were those previews which then end up with a recommendation to buy or not. I dont see why that is necessary at all. Isnt the idea that the reader makes up his/her own mind after reading the review? The problem with this recommendation is that a reader might have complete different expectations from a MMO then the reviewer has. This just asks for disappointed ppl. So I rather would see a pros/cons list instead with maybe some sections that are more general.

Something like this.

Graphics : A description about player toons maybe, armoursets, textures etc.

Animation : Is it choppy, does using a skill break off animations etc.

Music/Sound : What does it add to the game? Is it just generic music that you will soon turn off?

World : How immersive, guided paths?

PVP : (could break if up in types even if necessary or list it behind this) FFA, arena, RvR etc

Crafting/Gathering: recipedrops, nodehunting etc

Housing : Instanced or available at all

A list thats used with every review, with behind it a short description and then put in Pros if its exceptional, or Cons if its bad. Everything else just in seperate neutral section or so. If a summary would be made like this, then an overall score isnt even necessary. You can leave that up to the readers and their voting system. But anyone who wants to try a new MMO, finds a summary like this maybe more usefull.

Reviews for me serve the purpose of getting a inital overview of things, reading multiple site reviews and general player comments on a game is a good thing sure but the only review I would ever really trust is my own internal review - what i mean is the first month in a game will make a much bigger impact on potential future play than any review will

As far as rating are concenred, like the review themselves they are subjective and personal to the reviewer and I only take them lightly, getting included onto metacritic isnt such a big deal, most of us are wary of those scores now anyway so maybe running a system that doesnt conform to metacritics system would make you look more open/genuine (i am not saying to me you dont now)

So really, what ever you decide is fine with me :)