Please use this thread to post constructive suggestions as to our policies concerning reviews. How can we make reviews better for you?

I may have more suggestions later, but the one I wanted to get out at this moment is that of having 2 or 3 persons review a game. That way there is a greater chance of covering multiple playstyles, likes and systems. You can also structure the review with questions, kind of like an interview with multiple devs. Either you or another senior staff member asks the questions and all three give their take or lay out a list of certain game features and have them write a paragraph or two on them.

I think it should be anonymous, in that all involved not know the other is playing. That way they can't influence each other. As far as the numerical value, I'm not even near the world's most mediocre when it comes to numbers. Maybe an average of all three scores?

Either way, having one person give a review is a disservice to the games and their developers I think. Having a small group (2-3) of anonymous reviewers with varying systems would be a step forward, I think.

Edit: I can't spell paragraph, amongst other things...

Ha, kudos on this thread! What would improve reviews for me is:

General scoring guidelines for the reviewers. While this may seem like an obvious point, let me illustrate it with two examples of what I mean by "guidelines". Firstly, one could conceive guidelines that say something alike "motivate your score" in the sense that we should prevent minor issues with a game (such as performance issues on PCs which are on par or even below minimum system specs, hint hint) from being too influential on the final score. Secondly, you should be aware that scoring values differ per culture and by that I do not just mean a difference between X/5, X/10, and X/20 systems; a French student would probably marvel at a 14/20 in much the same way as a Dutch student would gloat at an 8/10, in turn much like how a German student will not find the same sense of accomplisment unless he has been graded a 9/10. Point being: although these scores differ proportionally, they are regarded with similar admiration, simply because French use high grades very sparingly and Germans think a 10/10 should always be attainable.

Have a review and perhaps the game itself looked at by more than one person in order to establish a more objective view on the game. This could be as simply as including a small box saying "Second Opinion" with that person's grade and their pros and cons in it.

Have the forum users grade a review. This could be as simple as a Dig/Bury system (comparable to the screenshot section, actually) for the reviews themselves, where uninformed players only have to look at the amount of Digs and Buries in order to avoid becoming MISinformed players. However, I would much prefer a scaled, X/10 grading system (for registered users only). The idea here would be that, once a review receives enough bad grades (say 5.0 or below on average, 20 or perhaps 50 grades minimum), the review is removed or at least very clearly (big, red, flashing neon letters) marked as being thought inaccurate by the vast majority of players.

Have consequences for reviewers who underperform. I can understand trustworthy reviewers are perhaps hard to come by, but it would serve the MMO community as a whole better when you post no reviews rather than flawed reviews (although this is obviously very indesirable). Make a clear line (for instance a 5.0/10 average review grade with a minimum of, say, 10 reviews) and if a reviewer's average is below this line, simply do not accept any more reviews from them.

Obviously the grades in the previous two points should be adjusted according to how players rate the reviews. A 'trial period' where the only function or grades is accumulating data on average grades given by players is a fine option, there.

For goodness sake, have a competent person spellcheck all reviews. What some reviewers manage to write up is absolutely appalling, containing unthinkable grammatical mistakes, obvious punctuation errors, and even incomplete sentences.

That's just my two cents, then. I furthermore think some form of supervising should be implemented due to the viability of the scores as listed in the Game List of MMORPG.com.

Originally posted by Khalathwyr

I may have more suggestions later, but the one I wanted to get out at this moment is that of having 2 or 3 persons review a game. That way there is a greater chance of covering multiple playstyles, likes and systems. You can also structure the review with questions, kind of like an interview with multiple devs. Either you or another senior staff member asks the questions and all three give their take or lay out a list of certain game features and have them write a paragraph or two on them.

I think it should be anonymous, in that all involved not know the other is playing. That way they can't influence each other. As far as the numerical value, I'm not even near the world's most mediocre when it comes to numbers. Maybe and average of all three scores?

Either way, having one person give a review is a disservice to the games and their developers I think. Having a small group (2-3) of anonymous reviewers with varying systems would be a step forward, I think.

 

Edit: I can't spell paragraph, amongst other things...

I also like the idea of have around 2 people do a review on a game. Few other thing make reviewers summit dxdiag report so you can be sure that they meet the system req, for the game that they are reviewing. Another thing might be if game servers are USA based make sure the person is located in the US, and not euro.
 

*Shakes fist at Khal* You posted my idea before I could. :p

I agree maybe 2-3 reviewers to catch any gross discreptancies due to latency or hardware as well as a better average.

You might want to add a extra week play time for reviewers since so many games can change after a certain level is attained or area reached, for good or bad. Also I don't know how you guys do it, but a certain minimum of playtime for a review should be enforced and have it posted on the article, ie "Based on 8 hours playtime/based on 20 hours playtime" etc etc.

 A couple things from me:

1. These are MMORPGs and they are made for long play. I would recommend that your reviewer play the game for at least 30 days. Preferably, make it to the level cap.

2. Have multiple people review the game. MMORPGs, unlike other games, supports multiple, very different, play styles.

3. Make sure all of their rigs meet the minimum specs.

4. Proofread and spell check each review twice over.

5. Have a senior staff member take all of the reviews, and combine them. The senior staff member cannot be one who played the game at all, and their opinion cannot get into the rewrite. Make all of the original reviews available in their full length, as well.

6. Maybe follow sites like IGN and don't review a game until it's been out for a month (IGN used to do this, I don't think they do now). This way what you're reviewing is a game that's not buggy and has been patched.

7. Make sure each reviewer plays the game at different settings. If they all play it at "Low" graphics, then you get a skewed result on the graphics.

8. Log the number of hours each reviewer plays and posts them on the review.

9. Include videos and screenshots.

10. Possibly also do a video review.

Thank You,

 - Eric

Hi guys,

MMORPG policy on reviews is clear, the reviewer awards the score.

This is fine if the reviewer takes into consideration all other relevant reviews on the site. As the fundamental reason for reviewing products side by side on one site is for readers to make an informed decision on what game to buy, play etc.

A lot of good suggestion have been made above on how to improve the quality of reviews, as not only are the scores seen to be inconsistent but the quality of the reviews are low.

MMORPG.com is considered by many as the definitive source of information on the genre, and attracts a hardcore fan base that demand this quality. At the moment it seems other sites are giving better reviews, Eurogamer, IGN not dedicated  MMO sites should not have better reviews than MMORPG.com! For the sake of the integerity of the site it is better to have no review than a bad review.

Originally posted by blakavar

*Shakes fist at Khal* You posted my idea before I could. :p

I agree maybe 2-3 reviewers to catch any gross discreptancies due to latency or hardware as well as a better average.

You might want to add a extra week play time for reviewers since so many games can change after a certain level is attained or area reached, for good or bad. Also I don't know how you guys do it, but a certain minimum of playtime for a review should be enforced and have it posted on the article, ie "Based on 8 hours playtime/based on 20 hours playtime" etc etc.

I'm just happy I'm not the only one who thinks it's a good way to go, lol!

I liked a lot of what you guys posted as well, especially the logging the hours played part.

 I would say 1 thing Jon,when you see a review that is clearly a bad call,bad review,or indeed like the FE review coming from a gimped system be big enough to say "hey we made a mistake and are pulling the review" then look into your mistake's and make a new review.

Mistake's happen all the time in life but the way you guy's ignore your mistake's make matter's worse...far more shame in ignoring the mistake than correcting it.

Nice one Jon.
Here my humble suggestions:

1) Do not allow people who follow the game to write the main review, this is the easy route, but not the wisest. (this will avoid the Aion incident, I still can't digest that 9.3)

2) Pick always a mix of 2-3 people who follow different genres to score the games........a 9 should only be given to a game who can be  enjoyed even by people who usually would not play that kind of game, in which case it is necessary to ask the opinion of people with different playstyles.

3) There should be a main reviewer with the other 2 people chipping in to correct or support the main reviewer. The main reviewer should write the article but all 3 needs to agree with the score (and as point 1, the main reviewer should never be someone who is following the game).

4) The reviewers should be able to put themselves in the shoes of the target audiance of the game.....reviewing Darkfall with the same measures you use to review WoW doesn't make sense, and of course will put DF in disavantage from the start.
"What DF doesn't have autoloot????" - There is a reason why DF doesn't have autoloot, the reviewer should be able to know the reason, instead of comparing it with the WoW system and coming to the conclusion that DF UI is poor designed.
That's a simple example to make, I hope it helps explaining what I mean..

5) The review should not be the reviewer personal opinion, he should be able to state his disliking for a particular feature, whilst at the same time acknowledge the same feature can be fun for another category of players (so they are still entitled to express their opinion as long it doesn't influence the final score)
Remember my(our) opinion is always better than anyone else (even the reviewer), so the reviewer needs to stick with objective points if he wants to convince me and the other players that he is talking sense.
If he states just his opinion, then he becomes just another bloke who is spouting rubbish(trash) on the net.

6) The system specs of the reviewer should always be way above the minimun requirements, but that should be a given.

7) If you get hundreds of posts disagreing with the review, you should have the guts to re-review the game by another team...........sometimes normal players are right, particularly in forums like this one where everyone has a decent experience in MMORPG and aren't exactly noobs who you can fool easily.

8) Reviewers have to have played at least halfway through the game (if it s level based), or they should have played for a month for an average of 30 hours a week, they should clearly state for how long did they play or which level they reached.

9) You need to award a bonus for innovation. In the MMORPG market is much needed nowadays. A game like Fallen Earth should be praised for the effort of doing something different. Aion should not be rewarded for being a WoW copycat (9.3?), not penalised either but certainly a game who brings nothing new to the genre should not be awarded a 9.

10) Reviewers should be realistic. They should take into consideration that a game developed by a small Indie will never be as polished as a game developed by a major publisher, for obvious reasons.
Of course I am not saying that a crappy game should be given the befit of the doubt if it is developed by an Indie, but an "expert" should be able to spot a game that has potential in the short/medium term , even if the release is a bit bumpy.

Lastly Jon, MMORPG.com is a marketing tool for developers whether you like it or not, so it is important you guys take responsability for your reviews.
A good or bad score on MMORPG.com can make or break a game, in particular if it's a small Indie like Icarus Studios or Aventurine.
It is paramount you guys take this review issue seriously, this is not an amateurish site anymore.
I do not read IGN for MMORPG news, I always check here first.
When you type MMORPG in Google this is the site that comes on top, so if someone is looking for MMORPG info in a way or another ends up here.
Hope you realise that (I know you do)

Originally posted by Khalathwyr

I may have more suggestions later, but the one I wanted to get out at this moment is that of having 2 or 3 persons review a game. That way there is a greater chance of covering multiple playstyles, likes and systems. You can also structure the review with questions, kind of like an interview with multiple devs. Either you or another senior staff member asks the questions and all three give their take or lay out a list of certain game features and have them write a paragraph or two on them.

I think it should be anonymous, in that all involved not know the other is playing. That way they can't influence each other. As far as the numerical value, I'm not even near the world's most mediocre when it comes to numbers. Maybe an average of all three scores?

Either way, having one person give a review is a disservice to the games and their developers I think. Having a small group (2-3) of anonymous reviewers with varying systems would be a step forward, I think.

 

Edit: I can't spell paragraph, amongst other things...

This, but don't average the reviewers scores together...Well, I guess you would have to with the way the scores are set up on other parts of the site.

Originally posted by ste2000

Nice one Jon.
Here my suggestions:

1) Do not allow people who follow the game to write the main review, this is the easy route, but not the wisest. (this will avoid the Aion incident, I still can't digest that 9.3)

2) Pick always a mix of 2-3 people who follow different genres to score the games........a 9 should only be given to a game who can be played even by people who usually would not play that kind of game, in which case it is necessary to ask the opinion of those people.

3) There should be a main reviewer with the other 2 people chipping in to correct or support the main reviewer. The main reviewer should write the article but all 3 needs to agree with the score.

4) The reviewers should be able to put themselves in the shoes of the target audiance of the game.....reviewing Darkfall with the same measures you use to review WoW doesn't make sense, and of course will put DF in disavantage from the start.

5) The review should not be the reviewer personal opinion, he should be able to state his disliking for a particular feature, whilst at the same time acknowledge the same feature can be fun for another category of players (so they are still entitled to express their opinion as long it doesn't influence the final score)

6) The system specs of the reviewer should always be way above the minimun requirements, but that should be a given.

7) If you get hundreds of posts disagreeng with the review, you should have the guts to re-review the game by another team...........you know sometimes normal players are right, particularly in forums like this one where everyone has a decent experience in MMORPG and aren't exactly noobs who you can fool easily.

8) Reviewers have to have played at least halfway through the game (if it s level based), or they should have played for a month for an average of 30 hours a week, they should clearly state for how long did they play or which level they reached.

9) You need to award a bonus for innovation. In the MMORPG market is much needed nowadays. A game like Fallen Earth should be praised for the effort of doing something different. Aion should not be rewarded for being a WoW copycat (9.3?), not penalised either but certainly a game who brings nothing new to the genre should not be awarded a 9.

10) Reviewers should be realistic. They should take into consideration that a game developed by a small Indie will never be as polished as a game developed by a major publisher, for obvious reasons.
Of course I am not saying that a crappy game should be given the befit of the doubt if it is developed by an Indie, but an "expert" should be able to spot a game that has potential in the short/medium term , even if the release is a bit bumpy.

Lastly Jon, MMORPG.com is a marketing tool for developers whether you like it or not, so it is important you guys take responsability for your reviews.
A good or bad score on MMORPG.com can make or break a game, in particular if it's a small Indie like Icarus Studios or Aventurine.
It is paramount you guys take this review issue seriously, this is not an amateurish site anymore.
Hope you realise that.

This mostly sums up my own thoughts. I have been unhappy with your reviews on many games. I'm also tired of hearing from reviewers than have not played many MMOs or computer game genres in general. Why? these people make comparisons to the few games they have played without knowing the full history of said feature or its use in other games or genres.

Also you need to be reviewing for a target audience, If your reviewer loves wow and Aion and thats all they have played much they probably wont have high thoughts about a indi game like Fallen Earth. This does NOT mean the reviewer should be a fanboy forum nerd of the game they a reviewing for the same reasons.

Lets get a little professionalism around here, its not a small site for people to shout out uniformed opinions anymore.

Originally posted by daylight01

 I would say 1 thing Jon,when you see a review that is clearly a bad call,bad review,or indeed like the FE review coming from a gimped system be big enough to say "hey we made a mistake and are pulling the review" then look into your mistake's and make a new review.

Mistake's happen all the time in life but the way you guy's ignore your mistake's make matter's worse...far more shame in ignoring the mistake than correcting it.

When I was talking about the mistakes, i wasn't talking about the score. I've explained my position on that. i was talking about the minimum requirements thing. Once I verified that he did indeed have the required specs, I re-posted the review.

I never said I was pulling it due to the score.

 Here's a copy-n-paste of a post I made from this thread (with an edit or two)... Consider it my two cents worth :-p....

In my humble opinion, I think mmorpg.com does need to seriously re-think how they present their reviews.... revise their review criteria, set an objective standard that every reviewer has to stick to - at least as closely as possible... *Something* to inject some kind of consistency into it.

I understand when Stradden says "different people review the games and have different opinions than other reviewers". That sounds great on its face. But... here's the problem...

When people see a review here, they see it as "MMORPG.com's review of XYZ game". When developer websites highlight reviews and comments and scores about their game, they cite "MMORPG.COM" as the source... not the name of the reviewer. The same would go for ads in magazines, on the back of retail boxes, etc.

Do you see what's going on? Every review that's put out by this site is seen as being the official opinion of mmorpg.com as a whole - not merely the person who wrote it.

I think it's time the folks at mmorpg.com realize this site's "grown up". You're not a small, upstart site like you were some years back. Your name (for better or worse) carries weight and anything you put out in terms of reviews is seen as being the collective opinion of the site... just like IGN's, or any gaming magazine.

I think this is why people react the way they do to such all over the place reviews... You're trying to cling to that "individuals' reviews" approach, but your website has grown way beyond that. It's far bigger than the sum of its parts. Beyond this site, MMORPG.com is a presence unto itself, it's beyond the individuals who edit or write for it. So when you defend the reviews by saying "well that's what those individuals think", it sounds very disingenuous, like a cop-out.

Again.. just my thoughts... but I think you guys and gals should seriously reassess the way you approach reviews and rate games. A good place to start might be what was suggested by someone else here.... do away with number scores. Those seem to create the most uproar - unless that's what you're after, of course.

 Get rid of unnecessary ratings.

I've seen it said or hinted at a few times, but I think it'd be a great idea to also add the playing history of the reviewers. It's quantifies their experience and give the viewer, and potential customer, some background on the person expressing their opinions about the piece. That way we can match our own gaming style preferences to theirs and see if they enjoy most of the things we do. If they do, then we may have a better idea of what a game has to offer us. If their past gaming experiences and preferences are vastly different, then that may spark our interest into trying the game as well.

I only have one suggestion

2 reviewers one with a high end set up,the second with a low end(still within specs)

That way we will have a better understanding of how our experiance might be compared to the reviewers

ie: low end reviewer has massive lagg/fps/studder but high end reviewer doesnt the game is not at fault.

Also we get two perspectives on the game and can better see the intent of the devs

Only thing I can suggest after all this...do not have another review by Joe.  He has little to no credibility on this forum anymore considering the travesty that MMORPG contributed to by not removing the problem and leaving it removed.

We don't know if he lied or not.  But why read more crap from him when we can choose not to.  His opinion is at odds with the folks involved and so is his validity in the subject matter based on how the entire matter was handled...poorly.

Retracting the review would have been the best choice.  Posting it again on the front page was a great big PISS OFF to the people who comprise your audience.

Frankly why would customers put up with that more than once?

My first suggestion would be get rid of the numbers all together.  Seriously they mean nothing it's just what lazy people use to "research" what game they want to play next.  My opinion is that reviewers are supposed to be communicators.  Let them do that let them tell me what they found that worked and what they found that didn't.  Let them communicate to me with their words what is what with the games without throwing on some random number between 1 and 10 that for all I know they got by rolling a 10 sided dice a couple of times.   Lets face it the last couple of reviews the words didn't measure up to the numbers presented at the end.  Heck reading the FE review and looking at the numbers I got the impresion the reviewer didn't play the game but cobbled together a bunch of press releases about the game and tacked on a bit about how bad the lag was at the end that they got off the Fallen Earth forums back at release and when asked to give a number thought of their favourite number 69 put a dot in the middle and called it a day.

Secondly if your going to review the games tell me a bit about yourself.  Reviewers need an up to date profile listing the games they play and have played so I know where your coming from preferably with a nice little link to said profile.  If you play Everquest alot then I'm going to take what you say about Mortal Online differently than if you play Ultima Online.  Conversly if you play DAOC I'm going to take what you say about Aion differently than if you play WOW.

I think thats about it.

There is no doubt that numbers are helpful. That's why they are so common. I think you should keep them. Equally helpful, as suggested in the previous post, is background on the reviewer. Multiple reviewers with shorter reviews is also good. I think 4 is an ideal number of reviewers to have.

The answer, believe it or not Jon, may lie in porn. :)

I am not kidding. Take a look at some of the award winning internet sites that are dedicated to reviewing the plethora of adult sites on the internet and you will see what I mean (you don't have to look at the pictures). The review process is truly amazing, and there is no doubt in my mind that any sort of site reviewing anything can learn a few things here.

I am not at all drawing a comparison between porn and mmo games, merely pointing out that publishing consistent, helpful reviews is a well trodden path and trying to add a little light humour while doing so :). The review process has undergone a long improvement process in all sorts of industries.

The current style of review on mmorpg is a bit like a newspaper publishing a book review. A book review is meant to be written in an entertaining way, and really does not pretend to be anything other than an opinion. In that case the quality of the author of the opinion is what really counts and people get to know book and film reviewers and either take notice of them or don't. You will note though that book reviews do not tend to give a mark out of 10.

In the end, I would say that you either allocate proper resources to get serious about doing consistent, helpful reviews or simply stop doing them.