Originally posted by Paragus1

 

 

 

 

 

 

 

2) Another idea in term of the timeline for a review would be to do 3 reviews as an example which could be numerical, but space them out over a time period, then use the average score of the 3 reviews to determine the final word.  Review the game on day 1, then on day 30, then on day 60 or 90 since this is usually the time frame where a lot of people may lose interest historically.

 

 

 

 

 

 

I like this idea. MMO's are impossible to fairly review too early into the lives however at the same time we want an early impression so it can help us to decide wether to invest in it. i know you do something similar to this with your thumbs up previews and your random re-reviews but as stated these seem to be random. if there was a set formula for you reviews ie: initial review in the first few week then as paragus states 2 more at later intervals, i think it would gve a much fairer review for an MMO. because of the evolving nature of MMO's, i dont think they can be reviewed the same way as standard games. A classic example of this would be your Aion review. The 1st review was one of the highest scores ive seen given out on this website and i agree that the game, at first, blows you away. but that soon fades which could have been reflected quite fairly in a 2 month review.

Originally posted by Slapshot1188

IMHO a non-scoring system is a cop-out.  It's a way of not having to grade bad games and "offend" anyone.

 

Just define review criteria which are constant for all reviews.. and weigh the criteria according to the values of the site.

I'd have to disagree. Some websites gives bad games a great score exactly because they don't want to offend anyone.

A non-scoring system simply allows players to read the review, learn about the game in larger detail and make a decision based on what they've read. Then again, that's only if the reviewer gives an accurate review and not gloss over bad features, but that's not a scoring system-related issue.

One small correction, if you're using tenths (like when you said, "What's the difference between a 6.6 and a 6.7?") you're not using a 1-10 scale per se, it's more like a 0-100 scale with a dot in the middle. That being said, since the quality of a game is so arbirtary, having a scale that fine doesn't help much. I would suggest usine a 1-5 scale with a +/- on rare occasions where a game is better than what's normal for the whole number rating but not quite good enough for the next one up. But make it rare. Keep the scale simple.

As I'm SURE has already been point out if you use a sliding scale 1-10 the AVERAGE gamer ignores most games below 7.  SO why waste a number?  That's just silliness.  Either the game is crap, it's playable, it's good, or ZOMG YOU HAVE TO PLAY THIS.  Honestly.

This review debate is silly.  Back in the day it was a 5 pt review system.  Then 10pt.  Now with the current 10 pt. system you're actually looking at 100 pts for a review (for example, 8.6 is really the 86% when you consider a review typically starts at .1, .2, .3, etc).  With all the games coming out, I can't afford to waste money on ANY games that get less than an 8.  But thats my personal sticking decision.

You just need to have a seperate scoring for different aspects of said MMO. I've seen a few reviews by other forumers that uses this type of scoring, and IMO it gives a complete picture of the reviewed MMO.

To be honest, the system that would work best in my opinion is a combination of the A to F ratings using multiple categories to obtain the final grade. Whether someone is American or not, the Grade system is pretty well known throughout the world (I live in New Zealand for crying)

That is quite honestly your best bet. Then maybe convert the grade to a number if you desperately have to for Metacritic, but if they can't rock a grade system, oh well. Cest la vie.

Taking out those massive pointless pictures that do nothing other than to make your 200 word articles look bigger would be a good start.

In a lot of ways review scores are saying either "buy", "try" or "avoid" the subject they are reviewing. Past that point, a lot of review systems are window dressing.

Personally I appreciate a review that has a short list of pros and cons around a title, plus I think you could also have a "Better than..." and "Worse than..." comparison (and that will still generate lots of site traffic as fans of all games come in to tell you how wrong you are!).

On the subject of bias: be open. Have a Bias Box in every review saying, "This review was written during Patch 1.0.1.1.3. We were provided with 5 complimentary copies of this title for review, the author is best friends with the lead developer and they had a beer every Friday on the company dime." Bias might exist, but be open about it so that readers know they aren't being misled. If they choose to disregard a review because of "bias", then I'm sure they will find a reason anyway.

Agree about needing different reviews for different periods. A Day One review, A Month One review and a Year One review could cover the different experiences / evolution of a MMO.

Why not use a system that professional analysts use for stocks, specifically the sliding bar of sell, hold, and buy.  But in a reviewing case, do not try, try the game out, must get NAO!!!  

Then with the use of multiple reviewers can get a fairly consistent and hopefully accurate review of the game itself across a relatively large sample.

i dont think having no scoring system at all is a good idea. ill be honest and state that unless its a game im extremely interested in, i'll rarely read a full review.  ill normally skim read or go straight to the bottom and read the conclusion and look at the score. im sure there are many more ppl out there who do  this. w/o an at glance review score.

i would probably start looking to other websites for my reviews. im lazy and dont have enough computer times to spend 15 mins reading an entire review

The problem, as I see it, is that it seems MMORPG.com doesn't really have any idea at all what makes a good MMORPG. Maybe some of the folks working here do...but the criteria you use to review, and the various things I've seen on this site (polls, blogs, etc) all make me think that your perspective on MMORPGs is dictated completely by the modern conventional wisdom in the genre. Sadly, the modern conventional wisdom leads to failed games or games with niche player bases. It's actually no surpise that the games you guys give the highest scores to end up losing most of their subscribers after three months. It's completely predicatable if you are looking at the right things.

Think of it this way... Say you are a site that reviews new automobiles. Say everyone in the industry is talking about the new paint colors, new door locks, and new interior fabrics that are being used in modern cars. Now say you base your review of new cars SOLELY on those things. Paint color, Interior fabric, and door locks.

Obviously, the score you gave to a car would very often be completely irrelevant, and not predictive at all of how successful that car would be in the market. Why? Because you aren't reviewing the things about a car that really matter. How reliable is it? How powerful and fuel efficient is the engine? How well do the breaks and steering work? Without reviewing the aspects of a car that really make it a good car...you can't possible write a useful review.

The exact same holds true for MMORPGs. You don't review based on the elements that really matter. In fact, you often don't even mention them...and if you do it's in a passing way that doesn't seem to end up affecting the overall score at all. Things like depth, challenge, replayability....the realism, internal consistency, and immersiveness of the world...the scope and size and ambition of the design. How invested does the game make you feel in the world and in your character? How much of a sense of danger or excitement or adventure does the game create moment-to-moment? How meaningful are the rewards (how much investment and risk are involved in obtaining them)? Is the game accessible, but still DEEP...are you still learning and mastering new and interesting elements even hundreds of hours into your play, or do you have the game down in the first 10 minutes and never really have to do anything different? What new elements and new ideas is the game useing to IMPROVE the above things over previous games?

The conventional wisdom in the post-WoW genre leads you to believe that all the wrong things are important. People have drawn the wrong (or at least incomplete) conclusions about what made WoW successful, and run with them as fast as they could. All we get now are watered down, ever-more-shallow, ever-quicker and simpler and less involved games. You're reviews are based on these exact criteria...so the worst offenders in terms of creating shallow and short and meaningless experiences end up getting the highest scores on your site. You might think of it this way (although it isn't a direct cause-and effect, it might as well be), developers are creating games specifically for your review style...instead of making good games.

If we go back to the car analogy, car manufacturers are now making cars with really awesome paint jobs and the very best fabrics in the interior...but they have extremely wimpy and inefficent engines, terrible brakes and steering, and are basically death traps. Since you only review the paint and the interior, you'd give these cars high scores....and that's exactly what you're doing. Giving games that are doomed to lose most of their players in a short time the highest scores as being the best the genre has to offer. It's not reallly that you're being intentionally deceptive, it's really just a matter of priorities and a deeper understanding of what makes a good MMORPG. This lame and over-simplified idea of "fun" which centers around instant gratification and constant reward...the idea that a good MMORPG is one you can play entirely solo and complete entirely in 15 minute intervals from your smart phone... It really just seems like you guys and gals don't have a clue of what this genre is supposed to be about, what sets it apart from other types of games, and what makes the core players in this genre excited and keeps them playing.

The timing of your review really shouldn't matter. I could tell you after a few hours of play whether or not an MMORPG is going to last. If you are looking at the right criteria, it's almost immediately apparent which games are created to be long-term virtual worlds that will draw in players for years, and which are just shallow little dances that get big box sales and then disappear. First you have to recognize what the genre should be shooting for...long-term truly social experiences where you make friends, build a community, and get wrapped up in the world for years. Where you have memorable experiences and meaningful rewards rather than loot pinatas and a forgettable grind. You have to agree on what an MMORPG really is...then you'll be able to spot the great games without much trouble, and you'll probably be able to provide a solid review even before the game launches.

Are things like polish, playability, smoothness, and accessibility important? Sure they are...and a truly great game is going to provide those things on top of the elements that really matter. You need to recogize that it's perfectly possible to have a great MMORPG that launches with some bugs...but it's impossible to have a bad MMORPG succeed in the mass market simply because it's polished.

If you need help figuring out what's important... just ask. I'm sure there are plenty of folks who would be glad to help, me included. =)

The only problem I have with the reviews is that they are way too generous. Professional critics have a job to do, they have a responsibility. If they say something is okay, they're making it okay.

Nowadays we have a slew of identical MMOs churned out to make a quick buck at launch, with almost no attempt for innovation. Just throw together some instances, battlegrounds and a gear grind. Done. Stop saying it's okay.

Having to review and MMO is darn near impossible. With all the different classes and combinations, and crafting classes how can anyone get a real idea of what the game fully offers (end game and all)? With that said here is my 2 cents:

When I read a review what am I looking for? Why would I the reader even bother reading the review unless I am trying to get something out of it. Here is what i am looking for:

1 - Is the game worth the purchase price (whatver it may be) and will it keep me entertained for the initial 30 days?

If I read a review and it can tell me the above, we are off to a good start. What does the game initially do well, what does it do poorly. Can I get on a server and play for more than 20 minutes without being kicked off for maintance? Is 30 days all I am going to need to get the most out of the game?

2 - Is the game worth the monthly fee moving forward?

Game could be great, but is it worth the $14.95 per month? Is there more to see that is worth 14.95 after the 30 days is up? This question is more tricky because instead of comparing it to other games charging the same price, there are lots of free MMO's out there are really good and cost nothing. How does it stack up to those games as well.

Also let's not assume just because it's an MMO and we are all fans of MMO's that just because it does one thing different or has a few different classes it's worth 14.95 a month. What does it do that hasn't been done before? What does it do that has been done a hundrend times before that it does better than anyone else? Is the world unique and worth the investment? Is the company committed to the product or is it a 2 month experiement to see if it hits?

3 - What are the different play styles if offers? Is it PvP centric, PvE centric, a REAL mix, does it have crafting?

All of these questions need to be answered, I have read reviews on other sites that have said games are a mix of PvP and PvE only to find the game was primarily PvP with little if any PvE content. I want to know if I like playing a melee class or range class what does the game offer to me? Is it solo, duo or group friendly or is it hardcore raid centric? is it a mix (been fooled here alot as well). Here is where to take fun into account. Each person has their own idea of what fun is, but here is where you can say if raiding is your thing and something you find fun, this game is for you. If you like to duo with the wife, maybe you stear clear of this. If your thing going solo is fun, maybe this isn't for you. this sort of information is very important.

4 - Going along with #3 what is the long term outlook with the game (taking company history into account)?

The last thing I want to do as a gamer is waste months building a character only to have the servers shut down and the game go belly up 6 months down the line. Does the company have any history behind it? What is the buzz on the game? Do the servers look alive? Or is this one of those games that after the first 30 days is up people will be packing up and leaving? As a fan and as people who review these games, it shouldn't be THAT difficult to get a first impression like that. Now granted, no game is going to have WoW's players base, but is the game sustainable?

These are the main points I look for in a review, the number/letter overall score really doesn't matter to me all that much. As you mentioned what is the difference in game that gets a 6.5,7.0 or a 7.5? Are they all worth 14.95 a month or just the 7.5?

If you wanted to give each catagory a seperate score than tally it up at the end, that could work. That way at least we could go to specifics and look for what is important to us. For example if the game is worth the 14.95 a month but had a rough launch, perhaps for #2 they get a good grade but for #1 they don't pending some updates or patches.

Again just my 2 cents.

Hi,

Thanks for asking for feedback, it's always great to see sites being well engaged with their community.  I put some thoughts together and had trouble getting them into a comment post without losing the thread - so decided to stick them in a presentation instead.

I can't embed the presentation here so hosted it on my own server: can access it <here>.

Good luck sorting out your approach - it's actually a pretty fun problem to be wrestling with imho.

Cheers,

BitJunkie

I made an extensive suggestion here, but wanted to stress the point that you need to use sub-genres. In other gaming sites people quickly see "Racing Game" or "Platformer" and the like and it sets their expectations. WIth sub-genres I mean "Sandbox", "Themepark" and the like, maybe you find better sub-genre names.

The burden (and joy) is usually on journalists to come up with genre names, hence I think you guys (and others) are still a bit clueless about your role. Grow up.

Nice post Stormwatch.

I have a couple of further thoughts:

1) MMORPG.com could setup an online questionaire form using their assessment criteria.  They could get people to take a bartle test as part of the questionaire and then use the result to rank the game for different player archetypes.

2) They could also work with Beta testers to do the same before launch with help from the developers.

3) MMORPG.com is in a unique position to hold developers to account and to be a useful source of information for development teams - if they set up 1) and 2) then they can use this information to bargain for involvement during development and so be a real "voice" for the community.

I think that behind the answer to the review  methodology lies a lot opportunity to do good and secure a key position in the development cycle.  If it was my organisation I'd be thinking a lot more strategically and figuring out ways to leverage such a committed readership.

Cheers,

BitJunkie

I think Fozzik nailed it right on the head. Too much emphasis is put into the wrong parts of the games and too much bonus is given to certain concepts that really are just function.  

Honestly when it comes down to it, I read reviews for sake of something to read. If I want to decide whether or not I'm going to play a game I check out as much information as I possibly can before hand. I've read some reviews that never give any information that wasn't already available.

First, thank you for reading our feedback on other topics to address the rating system.  Yes, the rating system used for MMORPGs is currently messed up.   Given the complexity of MMORPGs, here are some ways you can address the situation:

Give a grade score (A to F, N/A or Incomplete) for different aspects of the game (Graphics, PvE, PvP, Crafting, Content, Sound, Long Term Playability, ect.) and a final socre (1-10) for the initial preview and then another for full review. Clearly some of the grades for the game aspects can be either N/A (for example, game doesn't have PvP) or incomplete (meaning that it cannot be determined in the initial preview).  Then you can complete some of these grades when the final review comes out.  Players should also be able to vote on these aspects like we vote for the overall score of a game.

The benefit of such a system allows people to know how you feel about different aspects of the game and we know what you haven't had time to experience (content, PvP and Long Term Playability probably the main things incomplete on initial release review). Furthermore, we can figure out if the game is good at certain things.  As a PvP player, I'm more interested in the PvP score and Long Term Playability in an MMO, and less concerned about Sound, Content and PvE.  I'm sure others are more interested in PvE, Content, Graphics.   Right now, I have to search and read your reviews to learn about these issues, and it would be just easier if I could see a grade rating on these, then read up on it if it interests me.

As for the 1-10 scoring system, G4 TV uses the 1-5 system but I think they stated how the scores are granted (and could easily be applied to MMORPG.com).  A score of 1 (or 1-3 for MMORPG) was simply terrible and/or unplayable with bugs. Shadowbane's sb.exe error disaster around launch comes to mind as deserving of this score.   A score of 2 (or 4-5) is a bad game, or doesn't do what was advertised but doesn't crash because of bugs (e.g., Age of Conan perhaps at release since the PvP was broken and DX10 wasn't included and beyond Tortage the game lacked content).  A score of 3 (6-7) is average game, and MOST games should fall into this rating (e.g., RIFT, etc). A score of 4 (8-9) should be for exceptionally good games with lots of replayability and a score of 5 (10) should be the rare game that just rocks on all factors. 

The downside is that most games will have a score of 6-7, which right now MMORGP.com keeps most ratings at 7.0-8.9.  But I think having a more formal rating system would be better. If you read the re-review of AION at MMORPG.com, you'll see the difference. That re-review said the game was better than it was at launch, yet gave it a lower score ("because it feels like an 8.0" if I remember the quote). It was this that eventually turned me off to your scoring system, because it depends on the whims of the review and had no standard.  

This website's reviews on games that have PvP should be done by people who enjoy PvP (or allow people who enjoy PvP score that part of the game). I'm absolutely sick of hearing how the person doing the review doesn't like to PvP, and then goes ahead and discusses the good and bad of a game's PvP system. I don't remember exactly which review that was, but I read it on this site. I personally hate PvE and I know there is no way I could review that part of a game for people who enjoy PvE.

Finally, when a game receives a relatively high score, a follow-up review should be sooner if possible when the game changes rather quickly. DCUO gameplay changed DRAMATICALLY 1 month after the initial review, as population levels dropped dramatically making it a very different experience. Leaving the very positive review score unchanged despite a dramatically different game experience I thought was not professional. This website then had a follow-up article saying that everything was great except lack of population. Well, lack of population is a major problem, and it almost sounded like it was the players fault for not recognizing how great the game was. I believe that when a game's population declines dramatically after launch that received such a high score, there should be some kind of managerial control or intervention at MMORPG.com to see what went wrong in the review or what was missed, and if necessary correct the score in a re-review.  

I know this was a long post, but I like MMOs and I'd like a good rating system so I know what to buy and try and what to avoid.  Right now, I can't do that with the current system.

Originally posted by wildtalent

I like a combination of your ideas.  Use the letter system, as its easier to understand what constitutes an A over a B and then use the rating of different areas of a game and average those findings for a final letter grade.  Sort of like giving each game a GPA score.

this is exactly what I was thinking. Letter grade system is almost universally understood. Everyone has a general understanding of the difference between a A vs a B. But still make sure to incorporate aspects of the game into perhaps catergories giving an explanation of what you experience good or bad, so we know what to expect overall. I feel  that once you pass below a 5 the game is pretty much crap at that point, labeling it with a 1 or a 2 isn't going to really make the damage any worse so you do in fact end up with half a scoring system that is useless.

I would still go with full out reviews, though Previews are nice, but I feel a review is better as i want to know if the game is worth my intitial purchase, and not only that, will it be worth a continuing subscription or perhaps I should wait a couple months. The more informed I am, the better decision I can make about where to put my money.

Originally posted by Stormwatch

A couple of suggestions from my side.

Score

Stick to the 10 Point scale. As you say, it works well with MetaCritic and you want your reviews to appear over there as well. But [1] enforce consistency and assign each number with a [2] proper meaning, which you should also [3] spell out right next to the number or by providing a link to a rating policy page, where you may also conveniently explain your criteria. The reviewer should first have a verdict, only then translate it to the scale. Here is a take with keeping in mind that 5 should be the true average.

• 10 — Perfect
• 09 — Excellent
• 08 — Very Good
• 07 — Good
• 06 — Above Average
• 05 — Average
• 04 — Below Average
• 03 — Bad
• 02 — Atrocious
• 01 — Abysmal

 

Categories...

 

I just wanted to quote this poster because THIS (pay attention all Games Reviewers!) is how the average person views the one to 10 scale!

NOT 1-5 plus 5! that you guys use... (see the quote here from the original article):

Next, I wanted to talk about scoring systems. over time, I’ve become more and more annoyed with the 1-10 scoring system. The trouble is that numbers 1-5 are very rarely used because they represent a failing grade. For me to give any game a 5 or below means that it probably didn’t even run on my machine or it so wildly missed the mark that it doesn’t even resemble a running game anymore. So why, if it’s never or rarely used, are we taking up half of our scale to say the same thing: This game totally fails.

Then there are the degrees that a 1-10 scale creates. Who, other than the reviewer, really knows the difference between a 6.5 and a 6.6? How does that difference stay consistent from reviewer to reviewer? Heck, how does the scoring system stay consistent from reviewer to reviewer anyway?

I know that 1-10 is how most gaming sites do things, and many of them do it because it puts the score into a nice little package for MetaCritic. So I’m wondering, where MMORPG.com is concerned, how important is it that we stay on par with everyone else? Do you all enjoy comparing the numbers from one site to the next? Is that helpful for you as a consumer?

Well to us consumers a game that doesn't run = 0... not 5.

And don't be so scared of saying that a game "fails"... some of them really do!

A score of 1 says "Abysmal - was this game written / programmed by primary school kids with no talent?"

A score of 10 says "You MUST buy this game - because if you don't you will be a social lepper when everyone else in the world is playing it because, yeah, it really is that good."

And you may never give out 1s or 10s, again, so what?

Ah... but then it's not fair is it?  I mean a game that was an 8 in 2001 cannot be fairly compared to a game that is an 8 in 2011?  Well no.  So what.  I want to know the date of the review and I judge it based on that.  An 'average' 2011 game is bound to be vastly better than the original UO.  I expect that.  It's fine - really.

So, re-review the games that stick around every couple of years if there is a demand or you hear something has changed vastly improving the game.

Maybe even say "In 2005 this game was the trend setter with an overall score of 9/10 and 6 years on it's still a contender 7/10."

This is what we need to decide if we will actually invest our time and money.

If you want to break down by categories - please do - but overall score is where people look.

Other than that - some very good posts in this thread.