Originally posted by Yamota

For example, someone made a list on the SW:TOR forum about all the things this game was lacking. From open world PvP to guild functions to UI customizability to character customizability and found alot of things that are lacking, which are basic in most MMOs these days.

 

Ok, but most of that list was not something I liked, thought important or thought was a big deal.

So now how does one create a rating? I would have to hack out most of that list to make it palatable. Yet, for him, that list was his gold standard.

Once again, reviews are subjective.

Reviews should be based on what the game is trying to do and how well it does it. It should either be a letter grade (if we must have some sort of letter or number system) or only on a scale of one to five.

Or, maybe just a yay or nay. Give the reasons why the game is successful/fails and then your opinion of yay or nay. Then others can read what the reviewer said and make their own decisions knowing that the bullet points that the reviewer outlined were what led to ther decision.

So no the case of that SWToR list, that person would give a nay and we would know that those were his reasons. I could then look at his reasons and say "most of what he doesn't like are non-issues therefore I need to do more research.

One of the movie reviewers that I really like is "movie bob" on the escapist. Not because I always agree with his opinion but because he backs up his opinion with thought and examplse. It gives a good foundation for his "yay or nay" and that has greater meaning than listenign to someone bitch about something (or praise something) because they just "feel" it should be this way.

I have to agree with you Bill when you say that we should keep more attention to your (MMORPG staff) words then to the score you give. But sometimes your words aren't that good either. Even more important is that in every review you do, wether the words are "neutral" enough or not, the score is ultimately what EVERYONE sees and will compare the game to. AND scores are what will make new players get interested in the genre, not words. If they see a high score and see it as a fail, they will get interested in other stuff for a long time (or forever).

Loke666's way of scoring games is a good one and I took that idea from him before in other posts about scores.

If you, MMORPG staff, won't leave the numbers scoring system then why don't update it to 0-5 scoring system? You are an MMORPG dedicated staff so you KNOW this is true: games need real innovation and the genre is lacking progress, that can only become possible by listenning to their fans and possible clients opinions, following their ideas the best they can with something big and bold, and the most important thing is to fix the major flaws in the genre, like KS, griefing, etc. 

I think you have the capacity and the writers to fix your "flaws" too! To fix the score system and fix the things people like less/hate more in MMORPG site. Listen to us, try following the opinions of your clients and make  a scoring system where everyone will agree or slightly disagree, instead of keeping this one and having the same enormous quantity of bad "feedback" everytime you make a review.

It's true there are haters and lovers for games, but the scoring systems we have in todays main sites and forums are like wood to fire, the more you give the more it will burn.

I see some problems with the reviewing on this website:

If a reviewer loves the game (clearly noticable in the review), then he tends to make excuses for things missing or not good enough. He/She mentiones them, but it doesnt get reflected in the score of a category. This is because if you feel that the game is so tremendously fun and it should get a 9 for that, you tend to work back from that so you can give the respective categories also high scores.

If a reviewer doesnt seem to care for the game (review doesnt reflect either way) , these flaws get reflected in their scores. I really think that you guys should be more aware of this.

The reason for this imo, is that 'fun' isnt a category that can be treated objectively. But it does show that the used different categories (graphics, sound etc) dont make up how fun the game can be. This is why the endscore based on the different categories is nonsense and why it gets flamed so much.

So, while I think that the good and bad lists are a good idea, the endscore should go. Or it should be put seperately. By making sure that the reader understands that it isnt an overall score based on the different categories, but a score based on the reviewers personal experience (subjective). Otherwise it wont work.

Another reason for this is, that a lot of players dont give each category the same weight. Which also means that the endscore is useless for them.

So, please get rid of the overall endscore or turn it into a personal score from the reviewer. Then, list games in the gamelist based on overall reader's score. That average makes more sense for readers that just want to know the score, then the score from one reviewer. The readers that take the time to read the review care less about an endscore anyway, but probably more about the pro/con list and the review itself.

Originally posted by Sovrath

Originally posted by Yamota

For example, someone made a list on the SW:TOR forum about all the things this game was lacking. From open world PvP to guild functions to UI customizability to character customizability and found alot of things that are lacking, which are basic in most MMOs these days.

 

Ok, but most of that list was not something I liked, thought important or thought was a big deal.

So now how does one create a rating? I would have to hack out most of that list to make it palatable. Yet, for him, that list was his gold standard.

Once again, reviews are subjective.

Reviews should be based on what the game is trying to do and how well it does it. It should either be a letter grade (if we must have some sort of letter or number system) or only on a scale of one to five.

Or, maybe just a yay or nay. Give the reasons why the game is successful/fails and then your opinion of yay or nay. Then others can read what the reviewer said and make their own decisions knowing that the bullet points that the reviewer outlined were what led to ther decision.

So no the case of that SWToR list, that person would give a nay and we would know that those were his reasons. I could then look at his reasons and say "most of what he doesn't like are non-issues therefore I need to do more research.

One of the movie reviewers that I really like is "movie bob" on the escapist. Not because I always agree with his opinion but because he backs up his opinion with thought and examplse. It gives a good foundation for his "yay or nay" and that has greater meaning than listenign to someone bitch about something (or praise something) because they just "feel" it should be this way.

I would be totally down with that! Understated and doesn't take over the review.

Because it works so well I never paid much attention to it and simply when straight to the pros/cons list. Unless something scored a 1 or 2 out of 5 then I'd really pay attention why.

4.5 out of 5 for you, Sovrath. And everyone else using a similar system (deducting 5 from current scoring norm etc).

Originally posted by SBFord

@nblitz

Welcome to the American grading system employed at EVERY school in the country.

 

Just because it is used in american high schools doesnt mean that it's sensible, applicable to all situations or it has to make sense to the wide international audience you have here on the site. I mean that was a pretty ignorant thing to say and is pretty ignorant way of looking at things.

And the point of scores is that they are supposed to be an objective reflection of a game merits in  numbers, not  "yea there are flaws and it's unoriginal and bla bla but omgomg i LOOVVVEE this game and SW franchise, this game is so FUNNN!!11!!!".  Noone really cares about particular person's biased opinion about a game when they read a review. They want a professional and objective evaluation of the product.

This is not high school. There is a product made by a company for profits and people are wondering what's the deal with it and if they should spend their hard-earned money on it. So it is your responsibility as professionals in this business to be more serious about it and give an objective look at things.

Cheers.

Bill, your article came across as saying: "Read our words because we try to be honest, but dont look at the score. We all know its just BS."

I think we all anticipate a game receiving a 8.7 would be almost flawless. Not saying it cant have any signs of being a new fledged game, but it cant have much faults. This ISNT the case with SWTOR, and we all know it.

Giving SWTOR 8 in innovation, yet saying it isnt very innovative... That removes the scoring systems credibility right there.

JBFord saying "welcome to the USA high school scoring system" doesnt help your case much. What are you really saying? You like it dumb over there?

1-5 are hardly EVER given. Even the worst games dont go that low. Which obsoletes HALF of the scoring scale - leaving you with less wiggle room to score a game. Further more, 10 is NEVER given because it is a utopic score. Its been like this forever.

So on a scale seemingly from 1-10, you are in reality left with ONLY 6, 7, 8 and 9. Its not even a 5 point scale... 9 is in reality the highest you go, and giving swtor 8.7 strips away your credibility to a whole new level of nonsensical.

Your problem is

a) You get advertising money to pay your staff. Bias. Minus point for cred.

b) You cant backpeddal and change the review now without admission of guilt. It will probably be best for you to keep your current score on swtor. At least some of us will be able to keep our rosetinted glasses on a bit longer.

I don't understand the comparison to the American school grading system - there are good reasons you need at least a 60 to pass.  It's a percentage, and if you get below that, it means you're getting half the material wrong.  That is abysmal.  If you study, its actually not that hard to get over 90% - that basically just means you're paying attention.

Compare that to making a game, or any other form of entertainment.. and its really not comparable.  It's not a pass/fail system.  Getting a 6.0 doesn't mean the game is 40% wrong.  It really doesn't make any sense to grade games the way you grade school kids.

Most importantly, Consider the way the numbers are used.  A student needs to learn the material to progress, ideally doing so at as close to 100% as possible.  There's no reason to aspire to grade disparity.

An review rating system, OTOH, is more about comparing different products.  The more disparity, the better.  The more you cram all the ratings into a narrow top margin, the more meaningless they become, because they no longer function as a basis for comparison.

Bill,

Please don't micro manage the site...you're above that.

We live to snivel about the scores, nothing is going to change that. We read what you say, we just disregard it, and form an opinion for ourselves, and even if yours is similar to ours, you'll still be an idiot. Thats the only thing we can all really count on...not the scores...not the games... not the red light at the Krispy Kream... its the fact that we are all idiots, but we have opposable thumbs, so we got that much going for us.

VIVA LA SCORE-ITA-CHALUPA!!

No...really... don't touch the scores... or ...trunk monkey!

Originally posted by Vhaln

I don't understand the comparison to the American school grading system - there are good reasons you need at least a 60 to pass.  It's a percentage, and if you get below that, it means you're getting half the material wrong.  That is abysmal.  If you study, its actually not that hard to get over 90% - that basically just means you're paying attention.

 

Compare that to making a game, or any other form of entertainment.. and its really not comparable.  It's not a pass/fail system.  Getting a 6.0 doesn't mean the game is 40% wrong.  It really doesn't make any sense to grade games the way you grade school kids.

 

Most importantly, Consider the way the numbers are used.  A student needs to learn the material to progress, ideally doing so at as close to 100% as possible.  There's no reason to aspire to grade disparity.

 

An review rating system, OTOH, is more about comparing different products.  The more disparity, the better.  The more you cram all the ratings into a narrow top margin, the more meaningless they become, because they no longer function as a basis for comparison.

 

It's all right here. If reviewers don't understand why they need to match writing to score, or why there needs to be full use of the ten point scale instead of 8 - 10, I'm not sure there's any credibility to find in that particular professionals' perspective. I'm sorry, I really want to believe MMORPG.com is doing right by their readers and the work they author, but there needs to be a greater attention to providing an honest, credible, and evaluative score in reviews.

Originally posted by NBlitz

 

4.5 out of 5 for you, Sovrath. And everyone else using a similar system (deducting 5 from current scoring norm etc).

Darn, I always lose points in the swimsuit portion of the competition.

Originally posted by BillMurphy

Let me just say that I find it very funny that I ask you to put the scores secondary to the words, and the comments quickly became all about the scores.  :)

I get your concerns folks, really I do.  But we are trying to do our best service and honest recommendations on every game we review, which means we're writing about the good and the bad equally, which makes the scores seem unjust.  That's sort of my point.  But rather than us NOT write about the bad, I'm asking you to realize that a score of 8/10 can mean a game's a lot of fun despite its faults.

Additionally, the stance we take on scoring in general is akin to a Grade Scale in any American High School.  Anything below 6 is likely considered failing, a 7 is a C, an 8 is a B, and an A (90/100+) is damned near perfect. 

Again, you may not agree with this method, but it IS how we do things and we ARE always evaluating things.

 

I dont want to start anything close to a hate war with my question, but i do have to ask, and please say if you find me to be out of line.

Taking in consideration the rate system you just gave, can you please make the bridge between reviews mmorpg.com gave to games that were already sinking ships at the time of theirs launches (AoC, WAR, AION) by being full of bugs, not completed content wise, game design flaws, etc, and the recent review this same site gave to SWTOR?

Do we have to expect a possible thunderstorm eveytime a reviewed game doesnt get a perfect 10?

Originally posted by BillMurphy

Let me just say that I find it very funny that I ask you to put the scores secondary to the words, and the comments quickly became all about the scores.  :)

I get your concerns folks, really I do.  But we are trying to do our best service and honest recommendations on every game we review, which means we're writing about the good and the bad equally, which makes the scores seem unjust.  That's sort of my point.  But rather than us NOT write about the bad, I'm asking you to realize that a score of 8/10 can mean a game's a lot of fun despite its faults.

Additionally, the stance we take on scoring in general is akin to a Grade Scale in any American High School.  Anything below 6 is likely considered failing, a 7 is a C, an 8 is a B, and an A (90/100+) is damned near perfect. 

Again, you may not agree with this method, but it IS how we do things and we ARE always evaluating things.

Sorry, but the highlighted line is just not true. The reviews do contain some cons, but those are mostly light in weight and often give the impression that they are sort of placeholders just to have some cons at all.

The reviews barely ever touch the real problems of a game, those one can read about in the forums. It's just like all interviews with your industrie friends never contain the burning questions nor does your coverage ever display the many live problems a game has (like your at times daily Rift coverage never did).

Recent example the SWTOR review...who can deny that the game lacks numerous, a decade old basic MMORPG features, has bugs galore and suffers on conceptional design errors. Reviewed as an MMORPG, the list of cons could easily contain like 20 points.

But nevermind, we've got it by now. You try to do the split between those who pay you to write and those who just come to read.

If the scores dont matter, don't give any.

Do the writeup with full explanations of the testers feelings about features and let the readers decide. This site is about the written word and if gamers need a tl:dr on a review of a game they may actually have an interest in... maybe they should be watching videos on youtube.

Add in a precis about the reviewers typical gamestyle so the reader knows where the reviewer is coming from.

We don't need to be spoonfed, we're all past babyhood and have learned how to read.

Though sometimes.... I have to wonder.

Originally posted by VoIgore

But nevermind, we've got it by now. You try to do the split between those who pay you to write and those who just come to read.

 

Are you insinuating that publishers pay off reviewers? As much as I would have loved to get a giant stipend from Riot for my review I can say that your implication is 100% verifiably not true.

Also if the pros and cons list had 40 total points then a) no one would read the actual review and b) the review format would become unwieldy and messy.  If you want detailed pros and cons read the review in its entirety.  The small list is supposed to be a brief snapshot of some of the most prevalent problems/successes of the game, not an end-all-be-all judgement on the entire product and experience.

Heh, I remember being up at five in the morning after staying up all night tweaking my LoL review.  I knew putting that ten in Innovation could lead to being condemned in the comments (and that did happen to a point) but I'll be damned if I could dissuade myself from doing so.

Another reason I think Bill has a great point here is that sometimes the numbers are a bit misleading.  More than once I've gotten to the end of a review and averaged out the scores only to be faced with an overall score that I'm surprised at.  As tempting as it is to alter some category grades to bump the total score up or down I always left them as is as my category scores were my god's honest opinions.  That happened with the LoL review.

Also I like to make a point of giving more diverse scores in my reviews.  At the end of the day seeing a bunch of sevens and eights aren't very informative, which is why I prefer maning up so to say and doling out sixes and fives and even tens if I feel that is the case.

Maybe one day we can do away with numbers though, that would be grand! :]

If you are going to shell out a fair wad of cash to buy a game that you dont even read a review of, just look at the scores, send me some will you! :D

Geezus christ people, all reviews no matter which set of writers you're reading from is all indicitative of THEIR own experiences, perspectives, bias etc. It is there to help others, especially less knowledgable players regarding a specific field, provide some knowledge and background about things they aren't familiar with. No matter who writes them they are all going to be opinions, the people here aren't trying to run for President of Online Virtual Gaming Reality or anything so he's not here to represent the masses and ensure the laws/bills are written properly to represent the wants and desires of everyone around the world.

If you don't respect the opinion of the writers or have formed you're own a long time ago, no one is making you listen or follow it! If you think you're opinion so great, go get a job and become a writer so the world can hear your 2 cents that they most likely don't care about not nearly to the extent as others do at websites like these.  

If not, then stfu and take it for what it is. I appreciate the time and effort writers anywhere put into sharing their experiences and providing their opinions but in the end its merely their opinion. They aren't part of the MMORPG Standardized Scores for the World to Follow or Else, geezus man. When we are arguing over things like an "Excellent" because it reflects a high score OPINION of a writer and that becoming misleading, I think you as a gamer have turned a corner.

I can imagine many of you sitting at your desk with your monocle and top hat and puffy red sweaty cheeks typing away at how the writers are wrong again! When has game reviews become so political? Absolutely ridiculous.

---

Originally posted by sanman7890
Heh, I remember being up at five in the morning after staying up all night tweaking my LoL review.  I knew putting that ten in Innovation could lead to being condemned in the comments (and that did happen to a point) but I'll be damned if I could dissuade myself from doing so.Another reason I think Bill has a great point here is that sometimes the numbers are a bit misleading.  More than once I've gotten to the end of a review and averaged out the scores only to be faced with an overall score that I'm surprised at.  As tempting as it is to alter some category grades to bump the total score up or down I always left them as is as my category scores were my god's honest opinions.  That happened with the LoL review.Also I like to make a point of giving more diverse scores in my reviews.  At the end of the day seeing a bunch of sevens and eights aren't very informative, which is why I prefer maning up so to say and doling out sixes and fives and even tens if I feel that is the case.Maybe one day we can do away with numbers though, that would be grand! :]

---

In regards to utilizing numbers and scales as a way to express an opinion is that everyone takes these numbers and scale it differently to their own opinions. I highly doubt you can take 2 people, look at a set of scores, give them a scale and make them feel the same about them. A 5 to someone might mean neither good or bad, but to someone else it could just mean bad. A 10 might mean absolute perfection and near impossible to acquire to one person, but that could merely just mean that game was great (not necessarily perfect) to another.

Given this, I would support the notion to not utilize these numbers at all and just give like a word, "Terrible or Bad" or "Great or One of the Best" or "Mediocre, Fun but Needs Improvement" etc. This probably communicates more, keeps everyone more on the same page without the confusion of the numbers, which can mean different things to different people.

Originally posted by BillMurphy

Let me just say that I find it very funny that I ask you to put the scores secondary to the words, and the comments quickly became all about the scores.  :)





I get your concerns folks, really I do.  But we are trying to do our best service and honest recommendations on every game we review, which means we're writing about the good and the bad equally, which makes the scores seem unjust.  That's sort of my point.  But rather than us NOT write about the bad, I'm asking you to realize that a score of 8/10 can mean a game's a lot of fun despite its faults.



Additionally, the stance we take on scoring in general is akin to a Grade Scale in any American High School.  Anything below 6 is likely considered failing, a 7 is a C, an 8 is a B, and an A (90/100+) is damned near perfect. 



Again, you may not agree with this method, but it IS how we do things and we ARE always evaluating things.

 

There is a clear discrepancy between the scores given and the actual content of the reviews on this site. But I don't think the issue is the discrepancy itself, but the fact that we don't really know what those "scores" stands for. If each writer were to write the very same exact review, word for word, they would still score the game differently, and I think that's where members are confused.

I believe MMORPG.com should have a single grading system and have all the writers follow this same scale. Also explaining in detail the grading system and what each value stands for would be a good thing to do as well, so that everyone can be on the same "level" as far as reading and understanding. In addition, I believe Writers should also explain their scores, because as you said, a game can have many flaws yet be scored higher because it's a very enjoyable game.  If writers were to justify their scores, I believe it would be a lot easier for the readers to understand from the perspective of the writer.

I's also very difficult for me to read that scores don't really matter, yet when we look at the SWTOR Roundtable review (for example), the first thing you will read is this : "BILL MURPHY - 9/10". Or when you look at reviews from the review page of this website, "FINAL SCORE - X.X" with a relatively large police. These are the first thing that members will notice. If scores really don't matter as much as the reviews themselves, perhaps re-thinking the placement of those scores may be a wise thing.

The fact is that, scores do matters. Not everyone reads the entire review (as much as I believe that they should), so the scoring system is very important. It also reflects greatly in websites such as Metacritics. I think it was 2 years ago when one of the writers here (I believe it was Jon Wood?) wrote an article about reading comprehension if I remember correctly, and near the end of the article he asked members who commented to write something and include "Jon Wood" in the comment. Few did.

TL;DR Make an article about the grading score itself explaining each score in detail. Have writers follow this grading score more closely. Writers should explain the score they give.

We already have a criteria for reviews, and if we took all our time explaining our score there would be a dearth of writing about the actual game.

There are two sides to the coin unfortunately.