Jon Wood: The Story of the Week

It's that time of year again; time for the MMORPG.com Reader's Choice Awards. The premise is pretty simple: You come in and vote for your favorite game in a number of categories. This year, the MMORPG.com staff has selected nominees in each of our categories and it's up to you folks to vote on them.

Over the years, we've done the awards in a couple of different ways, using a number of different structures. I thought I'd take my column this week to explain why we do them the way that we do and why some of the other ways have been proven not to work.

Open Voting

A few times, we've tried simply opening the voting in each category up to every game on our Game List. At the time, we thought that this would be the fairest way to approach this kind of award. After all, everyone should be able to vote for anyone they want, right?

So, we painstakingly laid out all of the categories. We were thorough in how we described each one of them and offered helpful tips in things that people might want top consider when casting their vote in each of them.

Ok, so here's what went wrong: A few years back, we did this and EVE Online won EVERYTHING. I mean, EVERYTHING. Up to, and including categories where it just didn't make any sense. At the time, EVE Online had very very little in the way of PvE. Who won the best PvE award? Yup, EVE Online.

I guess we naively thought going in that a larger percentage of people would look at each of the categories, read our helpful hints and be done with it. We were wrong.

And I know what some of you are thinking right now: You're thinking that EVE Online's rabid fan base was the exception and that they turned out in droves to see their game conquer all. That just wasn't the case... or it was, but they weren't alone. In each category, the voting was roughly the same in terms of who came in second and so on down, regardless of the category.

What we learned: A large number of people don't care what the categories are, or the spirit of the awards, and will only vote for their favorite game. This method is flawed.

Editorial Awards

In any learning process, it's natural to see something fail miserably, and then run in the exact opposite direction in hoping to correct it next time, and that's exactly what happened.

After the open voting fiasco, we were a little bit embarrassed, so we mistakenly thought that the users were the problem, and that the only way to really fix our end of year awards, was to just give them out ourselves.

In theory, it's a great idea. We, a small number of people, can control who gets the awards. After all, the staff here make a living looking at, evaluating and learning about these games.

There are three fundamental flaws in this plan:

First, and this is a biggie: MMORPG.com is a player driven site. Its greatest strength is, and always has been, the community that visits it. Taking that out of the equation makes the place pretty hollow.

Second, because of the site's revenue comes from advertising, there will always be the perception that any awards handed out by our staff alone are bought and paid for. While this is never and has never been the case (in fact, the business end of the site is forbidden from speaking with editorial staff about site content), the perception still exists and being called a sellout really sucks.

Third, the six of us (and this is the biggest our editorial staff has ever been) do not make up a broad enough cross section of the larger MMORPG community to make such a decision.

In short, while this might be the simplest way for us to give out awards, it isn't the best.

Finally

So we finally decided on a hybrid model. We figured that by providing a number of nominees in each of our categories, and then opening it up for voting from the community, we could have the best of both worlds.

It's still not a perfect system. People will still vote only for their favorite game whenever that game appears as a nominee. We will still be accused of taking money for nominations. We will still make some mistakes.

That being said though, the voting is fair. More people consider each category carefully. Everyone has a voice, and we can be sure that whoever wins a particular awards at least qualifies for it.

We've learned a lot over the years and frankly, we're still learning.

So I hope that gives you a bit of a candid look into the thought process behind the way that we give out our awards. It's one of those things that, on the surface, really seems very simple. But in the end, if you want to be as fair as possible to everyone involved, there really is a lot more to it.